Efficient scalable coding concept

ABSTRACT

Scalable coding concepts are described. One aspect improves parallel decoding of inter-dependent layers of a multi-layer video data stream by introducing a long-term syntax element structure for guaranteeing that during a predetermined time period the pictures of the dependent layer are subdivided so that borders of the spatial segments of the pictures of the second layer and the spatial segments of the first layer overlay. Another aspect concerns upsampling from base layer to enhancement layer. Another aspect introduces a long-term syntax element structure allowing the decoder to determine the inter-layer offset for a predetermined time period. Another aspect introduces a type indicator field changing a way a layer indicator field within the NAL unit headers is to be interpreted. Another aspect allows different codecs/standards to be used for the different layers. Another aspect concerns a syntax element structure which indicates the inter-layer offset in units of the base layer blocks.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/123,184 filed Sep. 6, 2018, which is continuation of U.S.patent application Ser. No. 14/753,144 filed Jun. 29, 2015, which is acontinuation of International Application PCT/EP2014/050065 filed Jan.4, 2014, which claims priority from U.S. Provisional Application61/749,042 filed Jan. 4, 2013, and U.S. Provisional Application61/809,605 filed Apr. 8, 2013, all of which are incorporated herein byreference in their entireties.

BACKGROUND OF THE INVENTION

The present application is concerned with scalable coding concepts suchas scalable video coding.

Scalable coding concepts are known in the art. In video coding, forexample, H.264 involves an SVC extension (Scalable Video Coding)allowing a base layer coded video data stream to be accompanied byadditional enhancement layer data so as to increase the reconstructionquality of the base layer quality video in different terms, such asspatial resolution, signal-to-noise ratio (SNR) or the like. Therecently finalized HEVC standard will also be extended by SVC profiles.HEVC differs from its predecessor H.264 in many aspects, such as, forexample, suitability for parallel decoding/encoding and low delaytransmission. As far as the parallel encoding/decoding is concerned,HEVC supports WPP (Wavefront Parallel Processing) encoding/decoding aswell a tile parallel processing concept. According to the WPP concept,the individual pictures are segmented in a row-wise manner intosubstreams. The coding order within each substream is directed from leftto right. The substreams have a decoding order defined thereamong whichleads from the top substream to the bottom substream. The entropy codingof the substreams is performed using probability adaptation. Theprobability initialization is done for each substream individually or onthe basis of a preliminarily adapted state of the probabilities used inentropy coding the immediately preceding substream up to a certainposition from the left-hand edge of the preceding substream,respectively, on such as the end of the second CTB (Coded Tree Block).Spatial prediction does not need to be restricted. That is, spatialprediction may cross borders between immediately succeeding substreams.In this manner, such substreams may be encoded/decoded in parallel withthe locations of current encoding/decoding forming a wavefront whichruns, in a tilted manner leading from bottom left to top right, fromleft to right. According to the tile concept, the pictures are segmentedinto tiles and in order to render the encoding/decoding of these tiles apossible subject of parallel processing, spatial prediction across tileboundaries is prohibited. Merely in-loop filtering across tileboundaries may be allowed. In order to support low delay processing, theslice concept has been extended: slices are allowed to be switchable toeither initialize the entropy probabilities anew, to adopt the entropyprobabilities saved during processing a previous substream, i.e. asubstream preceding the substream to which the current slice beginbelongs, and to adopt the entropy probabilities having been continuouslyupdated until the end of the immediately preceding slice. By thismeasure, WPP and tile concepts are rendered more suitable for low delayprocessing.

Nevertheless, it would be more favorable to have concepts at hand whichfurther improve scalable coding concepts.

SUMMARY

An embodiment may have a video decoder for decoding a multi-layer videodata stream into which a scene is coded in a hierarchy of layers usinginter-layer prediction from a first layer to a second layer, the videodecoder supporting parallel decoding the multi-layer video data streamin spatial segments into which pictures of the layers are sub-divided,wherein the decoder is configured to inspect a long-term syntax elementstructure (e.g. tile_boundaries_aligned_flag) of the multi-layer videodata stream so as to interpret the long-term syntax element structureassuming a value out of a first possible-values-set (e.g.tile_boundaries_aligned_flag=1), as a guarantee that, during apredetermined time period, the pictures of the second layer aresubdivided so that borders between the spatial segments of the picturesof the second layer overlay every border of the spatial segments of thepictures of the first layer and periodically determine, in timeintervals smaller than the predetermined time period, the subdivision ofthe pictures of the first layer and the second layer into the spatialsegments based on short-term syntax elements (e.g. column_width_minus1[and column_width_minus1[i]) of the multi-layer video data stream, andif the long-term syntax element structure has assumed a value out of asecond possible-values-set (e.g. tile_boundaries_aligned_flag=0),periodically determine, in the time intervals smaller than thepredetermined time period, the subdivision of the pictures of the layersinto the spatial segments from the short-term syntax elements of themulti-layer video data stream such that, at least for a first possiblevalue of the short-term syntax elements, there exists a border betweenthe spatial segments of the pictures of the second layer not overlayingany of the borders of the spatial segments of the first layer, and, atleast for a second possible value of the short-term syntax elements,borders between the spatial segments of the pictures of the second layeroverlay every border of the spatial segments of the first layer.

Another embodiment may have a video encoder for encoding a scene into amulti-layer video data stream in a hierarchy of layers using inter-layerprediction from a first layer to a second layer so that the multi-layervideo data stream is decodable in parallel in spatial segments intowhich pictures of the layers are sub-divided, wherein the encoder isconfigured to insert a long-term syntax element structure and short-termsyntax elements into the multi-layer video data stream, the short-termsyntax elements defining, in time intervals, the subdivision of thepictures of the first layer and the second layer into the spatialsegments, and switch between setting the long-term syntax elementstructure to a value out of a first possible-values-set with, during apredetermined time period greater than the time intervals, setting theshort-term syntax elements to a proper subset out of a set of possiblesettings, the proper subset being chosen so that, during thepredetermined time period, the pictures of the second layer aresubdivided so that borders between the spatial segments of the picturesof the second layer overlay every border of the spatial segments of thefirst layer, or a value out of a second possible-values-set with, duringthe predetermined time period, setting the short-term syntax elements toany of the set of possible settings, the set of possible settingsencompassing at least one setting according to which a border existsbetween the spatial segments of the pictures of the second layer notoverlaying any of the borders of the spatial segments of the firstlayer, and at least another setting according to which borders betweenthe spatial segments of the pictures of the second layer overlay everyborder of the spatial segments of the first layer.

According to another embodiment, a method for decoding a multi-layervideo data stream into which a scene is coded in a hierarchy of layersusing inter-layer prediction from a first layer to a second layer, thevideo decoder supporting parallel decoding the multi-layer video datastream in spatial segments into which pictures of the layers aresub-divided, may have the steps of: inspecting a long-term syntaxelement structure (e.g. tile_boundaries_aligned_flag) of the multi-layervideo data stream so as to interpreting the long-term syntax elementstructure assuming a value out of a first possible-values-set (e.g.tile_boundaries_aligned_flag=1), as a guarantee that, during apredetermined time period, the pictures of the second layer aresubdivided so that borders between the spatial segments of the picturesof the second layer overlay every border of the spatial segments of thepictures of the first layer and periodically determining, in timeintervals smaller than the predetermined time period, the subdivision ofthe pictures of the first layer and the second layer into the spatialsegments based on short-term syntax elements (e.g.column_width_minus1[i] and column_width_minus1[i]) of the multi-layervideo data stream, and if the long-term syntax element structure hasassumed a value out of a second possible-values-set (e.g.tile_boundaries_aligned_flag=0), periodically determining, in the timeintervals smaller than the predetermined time period, the subdivision ofthe pictures of the layers into the spatial segments from the short-termsyntax elements of the multi-layer video data stream such that, at leastfor a first possible value of the short-term syntax elements, thereexists a border between the spatial segments of the pictures of thesecond layer not overlaying any of the borders of the spatial segmentsof the first layer, and, at least for a second possible value of theshort-term syntax elements, borders between the spatial segments of thepictures of the second layer overlay every border of the spatialsegments of the first layer.

According to another embodiment, a method for encoding a scene into amulti-layer video data stream in a hierarchy of layers using inter-layerprediction from a first layer to a second layer so that the multi-layervideo data stream is decodable in parallel in spatial segments intowhich pictures of the layers are sub-divided may have the steps of:inserting a long-term syntax element structure and short-term syntaxelements into the multi-layer video data stream, the short-term syntaxelements defining, in time intervals, the subdivision of the pictures ofthe first layer and the second layer into the spatial segments, andswitching between setting the long-term syntax element structure to avalue out of a first possible-values-set with, during a predeterminedtime period greater than the time intervals, setting the short-termsyntax elements to a proper subset out of a set of possible settings,the proper subset being chosen so that, during the predetermined timeperiod, the pictures of the second layer are subdivided so that bordersbetween the spatial segments of the pictures of the second layer overlayevery border of the spatial segments of the first layer, or a value outof a second possible-values-set with, during the predetermined timeperiod, setting the short-term syntax elements to any of the set ofpossible settings, the set of possible settings encompassing at leastone setting according to which a border exists between the spatialsegments of the pictures of the second layer not overlaying any of theborders of the spatial segments of the first layer, and at least anothersetting according to which borders between the spatial segments of thepictures of the second layer overlay every border of the spatialsegments of the first layer.

Another embodiment may have a computer program having a program code forperforming, when running on a computer, the inventive methods.

A first aspect of the present application is concerned with scalablevideo coding in connection with parallel processing concepts. Parallelprocessing concepts such as WPP and tile segmentation allow forparallely decoding the pictures of a video in spatial segments intowhich pictures are subdivided, e.g. in substreams, tiles or slices. Justlike spatial intra-picture prediction, inter-layer prediction limits thedegree of parallelization in decoding layers dependent on each other viainter-layer prediction. To be more precise, spatial intra-layerprediction limits the degree of parallelization in decoding the pictureof a single layer, which problem is solved in different ways. Forexample, when using tiles as spatial segments, spatial intra-layerprediction is restricted so as to not cross tile boundaries. In the caseof WPP substreams, the parallel processing thereof is performed in astaggered manner so as to result in the appropriate tilted processingwavefront. In the case of inter-layer prediction, the decoding of adependent layer is rendered dependent on co-located portions of areference layer. Accordingly, the decoding of a dependent layer'sspatial segment may be commenced at the earliest at the time where theco-located portion of the reference layer has already beenprocessed/decoded. The area of the “co-located portion” is enlarged inthe case of allowing for “motion compensation”, as is the case withinter-layer prediction in case of different views as different layers,or owing to upsampling from lower to higher layer. That is, it isfeasible for a video decoder supporting scalable decoding usinginter-layer prediction and parallel decoding to derive the degree ofparallelization in parallel processing inter-dependent layers fromshort-term syntax elements concerning these inter-dependent layers,which short-term syntax elements define the subdivision of the picturesof these inter-dependent layers into their spatial segments. However,steadily doing so is cumbersome and computationally complex. Beyondthat, when doing so, the video decoder is unable to appropriatelyschedule the decoding threads which run in parallel in order to decodethe multi-layer video data stream. Accordingly, in accordance with thefirst aspect of the present invention, parallel decoding ofinter-dependent layers of a multi-layer video data stream is improved byintroducing a long-term syntax element structure which, when assuming acertain value, guarantees to the video decoder that during apredetermined time period greater than the short-term syntax element'stime intervals, the pictures of the dependent layer are subdivided sothat borders between the spatial segments of the pictures of the secondlayer overlay every border of the spatial segments of the first layer.By this measure, the video decoder is able to rely on the fact that themulti-layer video data stream has been appropriately encoded so that thesubdivision of the inter-dependent layers' pictures into spatialsegments does not inadvertently reduce the degree of parallelizationfeasible among these inter-dependent layers. Rather, the decoder may,for the predetermined period, schedule the distribution of the spatialsegments onto the video decoder's parallel processing threads in advanceby exploiting the constraint that the spatial segments' borders in thedifferent layers overlay each other in the signaled manner. Still, thelong-term syntax element structure allows for switching off thisguarantee, thereby allowing, in other application scenarios or forhigh-end video decoders, to perform the parallel processing schedulingon a short-term basis depending on the short-term syntax elements only,i.e. without exploiting any guarantee concerning the relative locationbetween the borders of spatial segments of the inter-dependent layers.For purposes of decisions in opportunistic decoding, the long-termsyntax element may be used, too.

Another aspect of the present application concerns scalable codingaccording to which pictures of inter-dependent layers are subject tointer-layer prediction using upsampling from the base layer to theenhancement layer, in connection with parallel processing of theinter-dependent layers. In particular, this aspect concerns theinterpolation used to do the upsampling from base layer to enhancementlayer. Normally, such interpolation causes neighboring partitions of thebase layer picture to become dependent on each other. That is, theinterpolation causes that the interpolation result at the outercircumference of portions of the upsampled base layer reference pictureis dependent on both pixels/pels within the co-located partition of thebase layer picture as well as pixel s/pels of neighboring partitions. Inother words, the areas of the base layer picture serving as a referencefor inter-layer prediction for co-located portions to be predicted inthe enhancement layer picture “smear out” and widen. Inadvertently, suchinter-dependency caused by the interpolation of the inter-layerprediction negatively affects the degree of parallelism achievable inparallel processing of the inter-dependent layers. According to thissecond aspect of the present application, a syntax element is introducedwhich informs the decoder that the interpolation along the base layer'spartitions is modified so as to not mix-up pixels/pels of neighboringpartitions of the base layer picture, the partitioning of the base layerpicture and its upsampled version depending on the spatial segments ofthe enhancement layer picture or both the base and enhancement layers,for example. By introducing such a syntax element, the encoder is ableto switch between two modes: if the interpolation is restricted to leavethe portions of the base layer picture self-contained, i.e. restrictionis switched on, the degree of parallelism in parallel decoding of theinter-dependent layers which is maximally achievable is increased withslightly decreasing the interpolation quality along the partition edgesof the base layer picture, and in the case of not restricting theinterpolation, the parallelism degree is decreased, however, withincreasing the interpolation quality at the partition's edges.

A third aspect of the present application concerns scalable video codingwith parallel decoding of inter-dependent layers and seeks to alleviatethe decoder's burden of performing the parallel processing scheduling,i.e. distribute the spatial segments onto the parallel processingthreads, by introducing a long-term syntax element structure whichallows the decoder to determine the inter-layer offset—or inter-layerdelay—for a predetermined time period greater than the time intervals atwhich the short-term syntax elements signal sizes and locations of thespatial segments of the pictures of the inter-dependent layers as wellas the spatial sampling resolutions of these pictures. By introducingthe long-term syntax element signaling the inter-layer offset, the videoencoder is able to switch between two modes: according to a first mode,the encoder guarantees to the decoder a certain inter-layer offset,corresponding to a certain degree of parallelism between decoding theinter-dependent layers, and sets, within this predetermined time period,the short-term syntax elements correspondingly so that the actualinter-layer offset is equal to the guaranteed one or even lower. Inaccordance with the other mode, such a guarantee is not provided to thedecoder, and hence the encoder is free to set the short-term syntaxelements so as to meet other criteria such as, for example, optimallyadapting the short-term syntax elements to the video content during thepredetermined time period. The inter-layer offset, thus explicitlysignaled in the data stream, may be a count of base layer spatialsegments which at the minimum have to be decoded, before the firstspatial segment of the temporally co-aligned enhancement layer pictureis commenced with, when obeying this count over the whole predeterminedtime period, not facing any conflicts at least with respect to decodingthe first spatial segment of the enhancement layer pictures during thispredetermined time period.

A fourth aspect of the present application concerns scalable videocoding and the signaling of the layers to which the various NAL units ofthe multi-layer video data stream belong, as well as the location ofthese layers within a scalability space as well as the meaning of thescalability dimensions spanning the scalability space. In order to easethe tasks to be performed by intermediate network entities involved withtransporting a multi-layer video data stream, such information should beeasily accessible for these intermediate network entities and should beable to be managed easily. The fourth aspect of the present applicationis based on a discovery of the inventors, according to which the typicalapplication scenarios justify the expenditure for a type indicator fieldwhich changes away the layer indicator field within the NAL unit headersassociated with the various layers is to be interpreted: if the typeindicator field has a first state, mapping information in generalinformation NAL units maps possible values of the layer indicator fieldto the operational points and the layer's NAL units are associated withthe operational points using the respective layer indicator field andthe mapping information. This way, the mapping between layers andscalability constellations may be adapted variably and allows for amultitude of scalability spaces to be realized, with the drawbackhowever that the management overhead is increased. If the type indicatorfield has a second state, the layer indicator field is split into morethan one portion, and the operation point the respective NAL unit isassociated with is located by using the values of these portions ascoordinates of a vector within the scalability space. By this measure,the mapping between layers and scalability constellations allows for alower number of scalability spaces to be realized, with the advantage,however, that the management overhead for the network entities isdecreased. The layer indicator field may be the same in both casesirrespective of the application scenario, but the way the layerindicator field navigates the layer's NAL units through the scalabilityspace may be adapted to the current application and its specifics. Theadaptation advantages overcompensate the necessity of the additionalexpenditure for the type indicator field.

A fifth aspect of the present application concerns multi-layer videocoding, i.e. scalable video coding, allowing different codecs/standardsto be used for the different layers. The possibility of allowingdifferent codecs/standards to be used for successive layers enables abelated expansion of an already existing video environment to cope withmulti-layer video data streams which have been subsequently extended byfurther enhancement layers and to use, to this end, new and potentiallybetter codecs/standards. Network sinks not able to understandcodecs/standards of some enhancement layers are still able to handle thelower layers and multi-codec decoders are fed by a transport layerdecoder which identifies, for each NAL unit of the multi-layer videodata stream, which codec same is associated with and hands over the NALunits of the multi-layer video data stream to the multi-standardmulti-layer decoder accordingly.

A sixth aspect of the present application relates to multi-layer videocoding where both, pictures of a base layer and pictures of anenhancement layer are subdivided into an array of blocks, each. In thatcase, the inter-layer offset may be signaled efficiently by inserting asyntax element structure into the multi-layer video data stream, whichindicates the inter-layer offset for parallel decoding the pictures ofbase and enhancement layers in units of the base layer blocks. That is,the sixth aspect of the present application is based on the finding thatthe explicit transmission of a syntax element structure which indicatesthe inter-layer offset between base layer and enhancement layer in unitsof base layer blocks increases the data to be transmitted merely in aminor manner, whereas the computational complexity for the decoder isgreatly reduced compared to the case where the decoder otherwise derivesthe inter-layer offset for parallel decoding the pictures of base andenhancement layers on the basis of other syntax elements which reveal,for example, the block size of the base and enhancement layer blocks,the sample resolution of base and enhancement layer pictures and thelike. The sixth aspect is closely related to the third aspect whenrealizing the syntax element structure as a long-term syntax elementstructure so that the inter-layer offset is indicated to the decoder asa guarantee which holds true for a predetermined time period which isgreater than the time intervals at which the short-term syntax elementsin the multi-layer video data stream indicate those hints which would beotherwise necessitated in order to determine the inter-layer offset bycombining these syntax elements in a relatively complex manner.

Naturally, all of the above aspects may be combined in pairs, triplets,quadruples or all of them.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be detailed subsequentlyreferring to the appended drawings, in which:

FIG. 1 shows a video encoder serving as an illustrative example forimplementing any of the multi-layer encoders further outlined withrespect to the following figures;

FIG. 2 shows a schematic block diagram showing a video decoder fittingto the video encoder of FIG. 1;

FIG. 3 shows a schematic diagram of a_picture subdivided into substreamsfor WPP processing;

FIG. 4 schematically shows a video decoder in accordance with anembodiment according to which an inter-layer alignment of spatialsegments of base and enhancement layers is exploited so as to alleviatethe decoding process;

FIG. 5 shows a schematic diagram of a picture subdivided into codeblocks and tiles respectively, with the tiles being composed of integermultiples of the code blocks and a decoding order defined among the codeblocks following the picture's subdivision into tiles;

FIG. 6 shows a syntax example for realizing the embodiment of FIG. 4;

FIG. 7 shows a schematic diagram of a pair of base layer and enhancementlayer pictures differently subdivided into tiles;

FIG. 8 shows another exemplary syntax applicable in connection with theembodiment of FIG. 4;

FIG. 9 shows a schematic diagram of a picture, its subdivision intotiles and the application of an interpolation filter in order to performupsampling for the sake of inter-layer prediction;

FIG. 10 shows a schematic block diagram of a multi-layer decoder whichis configured to be responsive to a syntax element within themulti-layer data stream so as to switch on or off an upsamplinginterpolation separation;

FIG. 11 shows a schematic diagram of a pair of base layer andenhancement layer pictures using inter-layer prediction from base layerto enhancement layer using upsampling so as to transit from base layersample resolution to the increased enhancement layer sample resolution;

FIG. 12 shows a schematic diagram illustrating the switchable upsamplinginterpolation separation according to FIG. 10;

FIG. 13 shows a schematic diagram of an overlay of base layer andenhancement layer pictures, both being subdivided into WPP substreams;

FIG. 14 shows an exemplary syntax for realizing the embodiment of FIG.10;

FIG. 15a shows a schematic diagram of a spatially aligned overlay ofbase and enhancement layer pictures both being differently subdividedinto spatial segments;

FIG. 15b shows a schematic diagram of the overlay of base andenhancement layer pictures of FIG. 15a , but illustrating anotherpossibility for choosing a partitioning along which upsamplinginterpolation separation is performed;

FIG. 16 shows a schematic block diagram of a video decoder in accordancewith an embodiment according to which the video decoder is responsive toa long-term syntax element structure within the multi-layer video datastream so as to derive or not derive a guarantee concerning inter-layeroffset between base and enhancement layer decoding therefrom;

FIG. 17a shows a schematic diagram of a pair ofbase layer andenhancement layer pictures, both being subdivided into tiles, so as toillustrate the inter-layer offset signalization conveyed by thelong-term syntax element structure of FIG. 16 in accordance with anembodiment;

FIG. 17b shows a schematic diagram of a pair of base and enhancementlayer pictures, both being subdivided into substreams for WPPprocessing, so as to explain an example for implementing the long-termsyntax element structure of FIG. 16 in accordance with an embodiment;

FIG. 17c shows a schematic diagram of a pair of base and enhancementlayer pictures, both being subdivided into slices, so as to explain anexample for realizing the long-term syntax element structure of FIG. 16in accordance with an even further embodiment;

FIG. 18 shows a schematic diagram of a picture subdivided intosubstreams for WPP processing with additionally indicating the wavefrontresulting when parallel decoding/encoding the picture using the WPP inaccordance with an embodiment;

FIG. 19 shows a table illustrating the relation between minimuminter-layer decoding offset and block sizes and sampling resolutionratio between base and enhancement layers in accordance with anembodiment of the present application;

FIG. 20 shows an exemplary syntax for realizing the long-term syntaxelement structure signaling in accordance with FIG. 16;

FIG. 21 shows another example for a syntax for realizing the embodimentof FIG. 16;

FIG. 22 shows a syntax of a NAL unit header in accordance with anexample for an HEVC like configuration;

FIG. 23 shows a schematic block diagram of a network entity inaccordance with an embodiment according to which scalable coding isalleviated by allowing switching between different layer indicator fieldinterpretations;

FIG. 24 shows a schematic diagram illustrating the way of switching inresponse to a type indicator field;

FIG. 25 shows a schematic diagram illustrating the switchable layerindicator field interpretation in accordance with an embodiment infurther detail;

FIG. 26 shows an illustrative syntax for realizing the switchable layerindicator field interpretation in accordance with FIG. 23;

FIG. 27 shows a schematic diagram illustrating the layer indicator fieldswitching relating to the syntax of FIG. 26;

FIG. 28 shows a block diagram of a transport stream decoder configuredto simply discard enhancement layer NAL units of a codec other than thebase layer codec;

FIG. 29 shows a block diagram of a transport stream decoder interfacinga single standard multi-layer decoder, illustrating the transport streamdecoder's behavior in accordance with an embodiment;

FIG. 30 shows a transport stream decoder interfacing a multi-standardmulti-layer decoder and the transport stream decoder's behavior inaccordance with an embodiment;

FIG. 31 shows another example for a syntax for realizing the switchablelayer indicator field interpretation in accordance with a furtherembodiment;

FIG. 32 shows a schematic diagram illustrating a picture of any layer,subdivided into blocks with indicating a further subdivision of thepicture into spatial segments;

FIG. 33 shows a schematic diagram of a picture of any layer, subdividedinto blocks and tiles;

FIG. 34 shows a schematic diagram of a picture subdivided into blocksand substreams;

FIG. 35 shows a schematic block diagram of a video decoder in accordancewith an embodiment where the video decoder is configured to use syntaxelement structure in the data stream so as to derive an inter-layeroffset between inter-predictively processed pictures in units of blockshaving a raster scan order defined thereamong;

FIG. 36 shows a schematic diagram illustrating a possible mode ofoperation of the video decoder of FIG. 34 relating to the syntax elementstructure within the data stream in accordance with an embodiment;

FIG. 37 shows a schematic diagram illustrating a mode of operation ofthe video decoder of FIG. 35 in connection with a further embodimentaccording to which the inter-layer offset signaling is switchablebetween different explicit signaling types, i.e. signaling in units ofdifferent types;

FIG. 38 shows a schematic diagram illustrating a mode of operation ofthe video decoder of FIG. 35 in accordance with an even furtherembodiment according to which the inter-layer offset is continuouslysurveyed during parallel decoding of base enhancement layer pictures;

FIG. 39 shows a relationship between a rank of a certain block of apicture in accordance with a raster scan decoding order on the one handand row and column indices on the other in accordance with anembodiment;

FIG. 40 shows different examples for relative regular subdivisions ofbase and enhancement layer pictures into blocks and the consequencesresulting from these different examples;

FIG. 41 shows an example for a syntax for implementing any of theembodiments of FIGS. 35 to 40;

FIG. 42 shows another syntax example as an alternative to the one ofFIG. 41;

FIG. 43 shows a syntax example according to which the embodiments ofFIGS. 16 and 35 could be signaled at another portion of the multi-layerdata stream; and

FIG. 44 shows a schematic block diagram of a transport layer decoderinterfacing to a multi-layer multi-standard decoder in accordance withan embodiment.

DETAILED DESCRIPTION OF THE INVENTION

First, as an overview, an example for an encoder/decoder structure ispresented which fits to any of the subsequently presented concepts.

FIG. 1 shows a general structure of an encoder in accordance with anembodiment. The encoder 10 could be implemented to be able to operate ina multi-threaded way or not, i.e., merely single-threaded. That is,encoder 10 could, for example, be implemented using multiple CPU cores.In other words, the encoder 10 could support parallel processing but itdoes not have to. The bitstreams generated will also begeneratable/decodable by single-threaded encoders/decoders. The codingconcept of the present application enables, however, parallel processingencoders to efficiently apply parallel processing without, however,compromising the compression efficiency. With regard to the parallelprocessing ability, similar statements are valid for the decoder whichis described later with respect to FIG. 2.

The encoder 10 is a video encoder but in general the encoder 10 may alsobe a picture encoder. A picture 12 of a video 14 is shown as enteringencoder 10 at an input 16. Picture 12 shows a certain scene, i.e.,picture content. However, encoder 10 receives at its input 16 alsoanother picture 15 pertaining the same time instant with both pictures12 and 15 belonging to different layers. Merely for illustrationpurposes, picture 12 is shown as belonging to layer zero whereas picture15 is shown as belonging to layer 1. FIG. 1 illustrates that layer 1 mayinvolve, with respect to layer zero, a higher spatial resolution, i.e.,may show the same scene with a higher number of picture samples but thisis merely for illustration purposes only and picture 15 of layer 1 may,alternatively, have the same spatial resolution but may differ, forexample, in the view direction relative to layer zero, i.e., pictures 12and 15 may have been captured from different viewpoints.

The encoder 10 is a hybrid breed encoder, i.e., pictures 12 and 15 arepredicted by a predictor 18 and the prediction residual 20 obtained by aresidual determiner 22 is subject to a transform, such as a spectraldecomposition such as a OCT, and a quantization in atransform/quantization module 24. A transformed and quantized predictionresidual 26, thus obtained, is subject to entropy coding in an entropycoder 28, such as arithmetic coding or variable length coding using, forexample, context-adaptivity. The reconstructible version of the residualis available for the decoder, i.e., the dequantized and retransformedresidual signal 30 is recovered by a retransform/requantizing module 31and recombined with a prediction signal 32 of predictor 18 by a combiner33, thereby resulting in a reconstruction 34 of picture 12 and 15respectively. However, encoder 10 operates on a block basis.Accordingly, reconstructed signal 34 suffers from discontinuities atblock boundaries and, accordingly, a filter 36 may be applied to thereconstructed signal 34 in order to yield a reference picture 38 forpictures 12 and 15, respectively, on the basis of which predictor 18predicts subsequently encoded pictures of the different layers. As shownby a dashed line in FIG. 1, predictor 18 may, however, also, such as inother prediction modes such as spatial prediction modes, exploit thereconstructed signal 34 directly without filter 36 or an intermediateversion.

The predictor 18 may choose among different prediction modes in order topredict certain blocks of picture 12. One such block 39 of picture 12 isexemplarily shown in FIG. 1. There may be a temporal prediction modeaccording to which block 39 which is representative for any block ofpicture 12 into which picture 12 is partitioned, is predicted on thebasis of a previously coded picture of the same layer such as picture12′. A spatial prediction mode may also exist according to which a block39 is predicted on the basis of a previously coded portion of the samepicture 12, neighboring block 39. A block 41 of picture 15 is alsoillustratively shown in FIG. 1 so as to be representative for any of theother blocks into which picture 15 is partitioned. For block 41,predictor 18 may support the prediction modes just-discussed, i.e.temporal and spatial prediction modes. Additionally, predictor 18 mayprovide for an inter-layer prediction mode according to which block 41is predicted on the basis of a corresponding portion of picture 12 of alower layer. “Corresponding” in “corresponding portion” shall denote thespatial correspondence, i.e., a portion within picture 12 showing thesame portion of the scene as bock 41 to be predicted in picture 15.

The predictions of predictor 18 may, naturally, not be restricted topicture samples. The prediction may apply to any coding parameter, too,i.e. prediction modes, motion vectors of the temporal prediction,disparity vectors of the multi-view prediction, etc. Merely theresiduals may then be coded in bitstream 40.

A certain syntax is used in order to compile the quantized residual data26, i.e., transform coefficient levels and other residual data, as wellas the coding parameters including, for example, prediction modes andprediction parameters for the individual blocks 39 and 41 of pictures 12and 15 as determined by predictor 18 and the syntax elements are subjectto entropy coding by entropy coder 28. The thus obtained data stream 40as output by entropy coder 28 forms the bitstream 40 output by encoder10.

FIG. 2 shows a decoder which fits to the encoder FIG. 1, i.e., is ableto decode the bitstream 40. The decoder of FIG. 2 is generally indicatedby reference sign 50 and comprises an entropy decoder, aretransform/dequantizing module 54, a combiner 56, a filter 58 and apredictor 60. The entropy decoder 42 receives the bitstream and performsentropy decoding in order to recover the residual data 62 and the codingparameters 64. The retransform/dequantizing module 54 dequantizes andretransforms the residual data 62 and forwards the residual signal thusobtained to combiner 56. Combiner 56 also receives a prediction signal66 from predictor 60 which, in turn, forms the prediction signal 66using the coding parameter 64 on the basis of the reconstructed signal68 determined by combiner 56 by combining the prediction signal 66 andthe residual signal 65. The prediction mirrors the prediction finallychosen be predictor 18, i.e. the same prediction modes are available andthese modes are selected for the individual blocks of pictures 12 and 15and steered according to the prediction parameters. As already explainedabove with respect to FIG. 1, the predictor 60 may use the filteredversion of the reconstructed signal 68 or some intermediate versionthereof, alternatively or additionally. The pictures of the differentlayers to be finally reproduced and output at output 70 of decoder 50may likewise be determined on an unfiltered version of the combinationsignal 68 or some filtered version thereof.

In accordance with the tile concept, the pictures 12 and 15 aresubdivided into tiles 80 and 82, respectively, and at least thepredictions of blocks 39 and 41 within these tiles 80 and 82,respectively, are restricted to use, as a basis for spatial prediction,merely data relating to the same tile of the same picture 12, 15,respectively. This means, the spatial prediction of block 39 isrestricted to use previously coded portions of the same tile, but thetemporal prediction mode is unrestricted to rely on information of apreviously coded picture such as picture 12′. Similarly, the spatialprediction mode of block 41 is restricted to use previously coded dataof the same tile only, but the temporal and inter-layer prediction modesare unrestricted. The subdivision of pictures 15 and 12 into six tiles,respectively, has merely been chosen for illustration purposes. Thesubdivision into tiles may be selected and signaled within bitstream 40individually for pictures 12′, 12 and 15, 15′, respectively. The numberof tiles per picture 12 and 15, respectively, may be any of one, two,three, four, six and so forth, wherein tile partitioning may berestricted to regular partitioning into rows and columns of tiles only.For the sake of completeness, it is noted that the way of coding thetiles separately may not be restricted to the intra-prediction orspatial prediction but may also encompass any prediction of codingparameters across tile boundaries and the context selection in theentropy coding may also be restricted to be dependent only on data ofthe same tile. Thus, the decoder is able to perform the just-mentionedoperations in parallel, namely in units of tiles.

The encoder and decoders of FIGS. 1 and 2 could alternatively oradditionally be able to use the WPP concept. See FIG. 3. WPP substreams100 also represent a spatial partitioning of a picture 12, 15 into WPPsubstreams. In contrast to tiles and slices, WPP substreams do notimpose restrictions onto predictions and context selections across WPPsubstreams 100. WPP substreams 100 extend row-wise such as across rowsof LCUs (Largest Coding Unit) 101, i.e. the greatest possible blocks forwhich prediction coding modes are individually transmittable in thebitstream, and in order to enable parallel processing, merely onecompromise is made in relation to entropy coding. In particular, anorder 102 is defined among the WPP substreams 100, which exemplarilyleads from top to bottom, and for each WPP substream 100, except for thefirst WPP substream in order 102, the probability estimates for thesymbol alphabet, i.e. the entropy probabilities, are not completelyreset but adopted from or set to be equal to the probabilities resultingafter having entropy coded/decoded the immediately preceding WPPsubstream up to the second LCU, thereof, as indicated by lines 104, withthe LCU order, or the substreams' decoder order, starting, for each WPPsubstream at the same side of the picture 12 and 15, respectively, suchas the left-hand side as indicated by arrow 106 and leading, in LCU rowdirection, to the other side. Accordingly, by obeying some coding delaybetween the sequence of WPP substreams of the same picture 12 and 15,respectively, these WPP substreams 100 are decodable/codable inparallel, so that the portions at which the respective picture 12, 15 iscoded/decoded in parallel, i.e. concurrently, forms a kind of wavefront108 which moves across the picture in a tilted manner from left toright.

It is briefly noted that orders 102 and 104 also define a raster scanorder among the LCUs leading from the top left LCU 101 to the bottomright LCU row by row from top to bottom. WPP substreams may correspondto one LCU row each. Briefly referring back to tiles, the latter mayalso restricted to be aligned to LCU borders. Substreams may befragmented into one or more slices without being bound to LCU borders asfar as the borders between two slices in the inner of a substream isconcerned. The entropy probabilities are, however, adopted in that casewhen transitioning from one slice of a substream to the next of thesubstream. In case of tiles, whole tiles may be summarized into oneslice or one tile may be fragmented into one or more slices with againnot being bound to LCU borders as far as the borders between two slicesin the inner of a tile is concerned. In case of tiles, the order amongthe LCUs is changed so as to traverse the tiles in tile order in rasterscan order first before proceeding to the next tile in tile order.

As described until now, picture 12 may be partitioned into tiles or WPPsubstreams, and likewise, picture 15 may be partitioned into tiles orWPP substreams, too. Theoretically, WPP substream partitioning/conceptmay be chosen for one of pictures 12 and 15 while tilepartitioning/concept is chosen for the other of the two. Alternatively,a restriction could be imposed onto the bitstream according to which theconcept type, i.e. tiles or WPP substreams, has to be the same among thelayers. Another example for a spatial segment encompasses slices. Slicesare used to segment the bitstream 40 for transmission purposes. Slicesare packed into NAL units which are the smallest entities fortransmission. Each slice is independently codable/decodable. That is,any prediction across slice boundaries is prohibited, just as contextselections or the like is. These are, altogether, three examples forspatial segments: slices, tiles and WPP substreams. Additionally allthree parallelization concepts, tiles, WPP substreams and slices, can beused in combination, i.e. picture 12 or picture 15 can be split intotiles, where each tile is split into multiple WPP substreams. Alsoslices can be used to partition the bitstream into multiple NAL unitsfor instance (but not restricted to) at tile or WPP boundaries. If apicture 12, 15 is partitioned using tiles or WPP substreams and,additionally, using slices, and slice partitioning deviates from theother WPP/tile partitioning, then spatial segment shall be defined asthe smallest independently decodable section of the picture 12,15.Alternatively a restriction may be imposed on the bitstream whichcombination of concepts may be used within a picture (12 or 15) and/orif borders have to be aligned between the different used concepts.

Before discussing the above presented concepts of the presentapplication, again referring to FIGS. 1 and 2, it should be noted thatthe block structure of the encoder and decoder in FIGS. 1 and 2 ismerely for illustration purposes and the structure may also bedifferent.

According to a first aspect, which could be called “Alignment of TileBoundaries”, a long-term syntax element structure is used to signal aguarantee that, during a predetermined time period, such as a timeperiod extending over a sequence of pictures, the pictures 15 of thesecond layer are subdivided so that borders 84 between the spatialsegments 82 of the pictures of the second layer overlay every border 86of the spatial segments 80 of the first layer. The decoder stillperiodically determines, in time intervals smaller than thepredetermined time period, such as in units of individual pictures, i.e.in picture pitch intervals, the actual subdivision of the pictures 12,15 of the first layer and the second layer into the spatial segments 80and 82 based on short-term syntax elements of the multi-layer video datastream 40, but the knowledge on the alignment already helps in planningthe parallel processing workload assignment. The solid lines 84 in FIG.1, for example, represent an example where the tile boundaries 84 arecompletely spatially aligned to the tile boundaries 86 of layer 0. Thejust-mentioned guarantee would, however, also allow for the tilepartitioning of layer 1 to be finer than the tile partitioning of layer0 so that the tile partitioning of layer 1 would encompass further,additional tile boundaries not spatially overlapping any of the tileboundaries 86 of layer 0. In any case, the knowledge about the tileregistration between layer 1 and layer 0 helps the decoder in allocatingthe workload or processing power available among the spatial segmentsconcurrently processed in parallel. Without the long-term syntax elementstructure, the decoder would have to perform the workload allocation inthe smaller time intervals, i.e. per picture, thereby wasting computerpower in order to perform the workload allocation. Another aspect is“opportunistic decoding”: a decoder with multiple CPU cores may exploitthe knowledge about the parallelism of the layers to decide to try todecode or not try to decode layers of higher complexity, i.e. of higherspatial resolution or higher number of layers. Bitstreams that exceedthe capability of a single core might be decodable by utilizing allcores of the same decoder. This information is especially helpful, ifprofile and level indicators do not involve such indication on minimumparallelism.

In order to more clearly understand the just-outlined aspect of thepresent application, reference is made to FIG. 4 showing a video decoder600 which could be implemented as set out with respect to FIG. 2. Thatis, the decoder 600 is configured to decode a multi-layer video datastream into which a scene is coded in a hierarchy of layers usinginter-layer prediction from a first layer O to a second layer 1 asalready described above. The video decoder supports parallel decodingthe multi-layer video data stream in spatial segments into whichpictures of the layers are subdivided, the spatial segments being, forexample, tiles, WPP substreams or the like. In other words, the videodecoder is able to decode the multi-layer video data stream in paralleland in doing so, the video decoder 600 operates on the pictures 12 and15 of layers O and 1 in units of spatial segments.

For example, as outlined above, the spatial segments could be tiles andwhile video decoder 600 is configured to decode the pictures 12 and 15of layers O and 1 using intra-picture spatial prediction, the videodecoder 600 disrupts the intra-picture spatial prediction for each tileat its tile boundary. The subdivision of the pictures 12 and 15 into thetiles is signaled within the data stream 40 by way of short-term syntaxelements on a short-term basis, such as for example in units of timeintervals, such as for example individually for the time frames 604 towhich each picture 12 and 15 relates, i.e. for each pair of pictures 12and 15 belonging to a certain time frame 604. As described above, thesubdivision of pictures 12 and 15 into tiles could be restricted torectangular regular subdivisions only, i.e. into rows and columns oftiles. The short-term syntax elements 602 would thus set the number ofrows and the number of columns of the tile-subdivisioning for eachpicture 12 and for each picture 15 of both layers individually. Indecoding the inbound multi-layer video data stream 40, video decoder 600is configured to apply spatial prediction, and potentially temporalprediction. Optionally, video decoder 600 entropy decodes each of thetiles separately. If probability adaptation is used during decoding eachtile, video decoder 600 initializes the entropy probabilities for eachtile separately so that the tiles are entropy decodable in parallel. Inaddition to spatial prediction, and optionally temporal prediction, thevideo decoder 600 supports inter-layer prediction as far as the decodingof the tiles of pictures 15 of layer 1 is concerned. As described above,inter-layer prediction may concern different parameters involved indecoding layer 1: inter-layer prediction may predict a predictionresidual of layer 1, such as transform coefficients, prediction modesused in decoding layer 1, prediction parameters used in decoding layer1, samples of the enhancement layer 1 picture and so forth. Theinter-layer prediction predicts portions within tiles of pictures 15 oflayer 1 based on already decoded portions of pictures 12 of layer0—either directly (centrally) co-located ones or ones slightly spatiallydeviating from the directly co-located position by way of, for example,a disparity vector prediction parameter controlling the inter-layerprediction in the case of the layers 0 and 1 concerning, for example,different views of the same scene.

Video decoder 600 is responsive to a long-term syntax element structureof data stream 40, indicated using reference sign 606 in FIG. 4, so asto differently process a predetermined time period 608 following thelong-term syntax element structure 606, the predetermined time period608 encompassing several time intervals, i.e. multiple time frames 604for which the short-term syntax elements 602 individually signal thepicture's subdivision into tiles. It should be noted that 608 may relateto the scope (=time period) of an SPS and SPS changes lead tosignificant re-initialization anyway. The just-mentinoed note is alsovalied for all embodiments relating to the other aspects as far as thelong-term feature is mentioned therein. In particular, if the long-termsyntax element structure 606 assumes a value out of a firstpossible-values-set, the video decoder 600 interprets this circumstanceas a guarantee that during the predetermined time period the pictures 15of layer 1 are subdivided so that borders between the tiles of pictures15 overlay every border of the tiles of the pictures 12 of layer 0. Thevideo decoder 600 still inspects, in that case, the short-term syntaxelements 602 in order to determine for the time intervals 602 within thepredetermined time period 608 the subdivision of pictures 12 and 15 intotheir tiles, but video decoder 600 may rely on the fact, and willrealize that, the borders of the base layer tiles of pictures 12 arecompletely overlaid by borders of the enhancement layer tiles ofpictures 15, i.e. that the tile subdivision of pictures 15 locallycorresponds, or represents, a spatial refinement of the subdivision ofpictures 12 into tiles, by comparing each time-aligned pair of pictures12 and 15. As described above, the video decoder 600 may take advantageof such signaling, i.e. long-term syntax element structure 606 assumes avalue out of the first possible-values-set, by accordingly schedulingthe parallel processing of the tiles of pictures 12 and 15 withinpredetermined time period 608 in parallel, i.e. in parallel decoding oftiles of a temporally aligned pair of pictures 12 and 15. For example,in case of the long-term syntax element's structure assuming the valueout of the first possible-values-set, the video decoder 600 may know thefollowing: for a certain picture 12 of layer 0, the first tile in tileorder among the tiles of picture 12 either locally coincides with arespective tile of the time-aligned enhancement layer picture 15, orcompletely locally overlays the first tile of the time-alignedenhancement layer picture 15 in tile order among the tiles of theenhancement layer picture 15. Accordingly, at least in the case ofinter-layer prediction without disparity/motion compensation, videodecoder 600 may commence decoding the first tile of the enhancementlayer picture 15 as soon as the decoding of the first tile of thetime-aligned base layer picture 12 has been finalized, since thejust-mentioned guarantee indicates to the video decoder 600 that theco-located portions of the base layer picture 12 needed for inter-layerprediction will be available for the whole first tile of the enhancementlayer picture 15. The inter-layer offset or degree of parallelismbetween base layer pictures 12 and enhancement layer pictures 15 maythus be recognized/determined by video decoder 600 to be equal to onetile of the base layer pictures 12. The offset may be slightly increasedin case of the inter-layer prediction involving disparity vectors havinga non-zero vertical component and/or disparity vectors having ahorizontal component which shifts the corresponding portions within thebase layer picture towards the right, wherein the tile order among thetiles may lead in a raster scan order row-wise from the left top cornerof pictures 12, 15 towards the bottom right corner thereof.

If the long-term syntax element structure, however, assumes a value outof a second possible-values-set, which is distinct from the firstpossible-values-set, video decoder 600 does not take advantage of anyguarantee, but plans and schedules on a short-term basis using theshort-term syntax elements 602 the parallel decoding of the tiles ofpictures 12 and 15 with, potentially, parallel decoding for at leastsome of the time-aligned pairs of pictures 12 and 15, tiles of base andenhancement layers. In that case, the video decoder 600 determines theminimum inter-layer offset or interlayer spatial processing offset inthe parallel decoding between layers 0 and 1, i.e. the degree ofparallelism between layers 0 and 1, on the short-term basis which is,however, a cumbersome procedure. At least for a subset of the set ofpossible values of the short-term syntax elements, there exists a borderbetween the spatial segments of the pictures of the second layer notoverlaying any of the borders of the spatial segments of the firstlayer. But a further subset of the set of possible values for theshort-term syntax elements exist according to which borders between thespatial segments of the pictures of the second layer overlay everyborder of the spatial segments of the first layer. The latter subset issolely used in case of the long-term syntax element indicating tileboundaries alignment between base and enhancement layers.

Additionally or alternatively, video decoder 600 may use or exploit thefact that the long-term syntax element structure assumes a value out ofthe first possible-values-set in order to perform a trial of, i.e. tryto perform, decoding layer 1 at all, with refraining to perform thistrial in case of the long-term syntax element structure 606 assuming avalue out of the second possible-values-set. In that case, especiallyfor battery driven devices, valuable computation power is saved in caseswhere the outcome or success in decoding the enhancement layer 1 intime, i.e. in real time, is speculative. Is it worth mentioning thatrefraining may also be chosen based on the level indicators mentionedbelow with respect to the fourth aspect.

Although FIG. 4 has been described above exemplarily using tiles asspatial segments, it is clear that the video decoder 600 may takeadvantage of the long-term syntax element structure and the guaranteewhich may be signaled thereby, in connection with other spatial segmentssuch as substreams or slices. In the former case, the video decoder 600would decode the pictures 12 and 15 of the layers using intra-picturesspatial prediction with decoding the spatial segments of a picture ofthe first layer 12 in parallel and supporting the intra-picture spatialprediction crossing boundaries of the spatial segments of the picture ofthe first layer and with obeying a decoding delay between decoding ofthese spatial segments, i.e. substreams. As described above, substreamsmay correspond to horizontal stripes of the respective picture, i.e.vertically subdivide the respective picture. In decoding each substream,video decoder 600 may use a decoding order which generally leads fromleft to right, and the decoding order defined among the substreams of apicture may lead from top to bottom. Using a typical spatial predictionconcept, according to which spatial prediction is performed from a topneighboring already decoded portion and a left-hand already decodedportion of the current picture, obeying a certain decoding delay betweenimmediately succeeding substreams, thus, allows a parallel decoding ofthe substreams. The decoding delay may be measured in, for example,units of LCUs. This may be done in pictures 12 of layer 0 as well aspictures 15 of layer 1. Thus, parallelism in decoding the video datastream may involve both parallelism within the pictures 12 and 15individually, but also the parallel decoding of substreams belonging topictures 12 and 15 of different layers of one time frame 604. As far asan optional entropy decoding of the substreams is concerned, same mayinvolve an adaptation of the entropy probability during the course ofdecoding the respective substream. The first substream in substreamorder of each picture 12 or 15 may be subject to an individualinitialization of the entropy probabilities independent from othersubstreams. Any subsequent substream may be subject to entropyprobability initialization depending on intermediately adapted entropyprobabilities of the immediately preceding substream in decoding orderof the same picture such as by adopting the entropy probabilitiesadapted during decoding the immediately preceding substream up to acertain distance from the left-hand of the respective precedingsubstream, such as after having decoding two LCUs of the immediatelypreceding substream.

Even in the wavefront parallel processing substream case, the videodecoder 600 is able to take advantage of the long-term syntax elementstructure 606: if the guarantee is signaled by way of this syntaxelement structure 606, video decoder 600 may rely on the fact that allborders between consecutive/neighboring substreams of the base layerpictures 12 within the predetermined time period 608 are overlaid by arespective border between neighboring/successive substreams of thetime-aligned enhancement layer pictures 15. That is, either a base layersubstream locally coincides with a respective enhancement layersubstream of a time-aligned enhancement layer picture 15, or sameexactly corresponds to two or more substreams of the time-alignedenhancement layer picture. Accordingly, if the guarantee applies, thedecoder 600 knows that decoding the first substream of the time-alignedenhancement layer picture 15 may be commenced as soon as the decoding ofthe first substream of the base layer picture 12 has been finalized.

As described above, differing from the tile subdivisioning, theshort-term syntax elements 602 may be selected such that same define theposition of the substreams in pictures 12 and 15 in relation to asubdivision of these pictures into some coding blocks, such as LCUs.Substreams may accordingly be a collection of one or more rows of suchcoding blocks. As in the case with the tile subdivisioning, the timeintervals 604 may be such that the short-term syntax elements 602 signalthe subdivision of pictures 12 and 15 into substreams on a per picturebasis, i.e. on a per picture frame 604 basis. If the guarantee is notprovided by the long-term syntax element structure 606, video decoder600 may nevertheless try to decode, in parallel, substreams of differentlayers of a common time frame, but in order to do so, the video decoder600 needs to inspect the short-term syntax element 602.

As it is the case with using tiles as spatial segments, the videodecoder 600 may render a speculative trial of decoding the enhancementlayer 1 dependent on the value assumed by the long-term syntax elementstructure 606.

It should be noted that as far as the video encoder side is concerned, arespective video encoder which may be implemented as shown in FIG. 1,inserts and sets the long-term syntax element structure 606 into datastream 40 and decides on granting the guarantee to decoder 600 or not.If granted, the encoded restricts the possibilities in setting theshort-term syntax element 602 for the corresponding predetermined timeperiod 608 to settings which obey the border alignment guarantee. Ifnot, the encoder remains free to set the short-term syntax elements 602during time period 608 as it likes to. When using tiles as the spatialsegments, the encoder obeys the restriction that the spatial predictiondoes not cross tile boundaries and that the optional entropy encoding ofthe tiles of the pictures 12 and 15 is performed in a self-containedmanner for each tile. For example, the entropy probabilities areinitialized for each tile anew independent from other tiles, for eachtile. In case of substreams, the entropy probability initialization ofthe substreams is performed anew, i.e. independent from any othersubstream, for any first substream of a respective picture 12, 15, andby adopting the entropy probabilities as adapted up to an intermediateposition of an immediately preceding substream as far as any second andfollowing substreams are concerned. The spatial prediction is performedwithout any restriction concerning the crossing of substream borders.

In a manner described with respect to FIG. 4, the alignment conceptcould be introduced into the currently envisaged extension of the HEVCstandard, namely in the manner described below. Insofar, the descriptionbrought forward immediately in the following shall also be interpretedas a basis of possible implementation details concerning the descriptionbrought forward above with respect to FIG. 4.

HEVC allows dividing the CTBs of a coded base layer picture via a gridof vertical and horizontal boundaries into rectangular regions that arereferred to as tiles and can be processed independently except forin-loop filtering. The in-loop filters can be turned off at tileboundaries to make them completely independent.

Parsing and prediction dependencies are broken at tile boundaries muchlike on picture boundaries, whereas in-loop filters can cross tileboundaries if configured accordingly in order to reduce tile boundaryartifacts. Therefore, processing of individual tiles does not rely onother tiles within a picture completely or to a vast extent depending ofthe filtering configuration. A restriction is installed in that all CTBsof a tile should belong to the same slice or all CTBs of a slice shouldbelong to the same tile. As can be seen in FIG. 1, tiles force the CTBscan order to regard the order of tiles, i.e. going through all CTBsbelonging to the first, e.g. upper-left tile, before continuing with theCTBs that belong to the second tile, e.g. upper-right. Tile structure isdefined through number and size of the CTBs in each tile row and columnthat constitute a grid within a picture. This structure can either bechanging on a per frame basis or stay constant throughout a coded videosequence.

FIG. 5 shows an exemplary division of CTBs within a picture into ninetiles. The thick black lines represent tile boundaries and the numberingrepresents the scanning order of CTBs, also revealing a tile order.

An enhancement layer tile of an HEVC extension can be decoded as soon asall tiles are decoded that cover that corresponding image area in thebase layer bitstream.

The following section describes constrains, signaling and decodingprocess modifications that allow easier access to the base layerinformation using the concept of FIG. 4.

The simplest case for tile level parallelization is, when tileboundaries in base and enhancement layer are aligned. For SNRscalability this means, that the boundaries are exactly at the sameposition. For spatial scalability this means, that for each twoenhancement layer pels, which belong to the same tile, the correspondingbase layer pels also belong to the same tile and that that for each twobase layer pels, which belong to the same tile, the correspondingenhancement layer pels also belong to the same tile.

HEVC features short-term signaling corresponding to 602 in FIG. 4 toindicate the dimension and structure of tiles within a picture on a perpicture basis by the use of picture parameter sets given below withcolumn_width_minus1 [i] and row height minus1 [i] from [1]. FIG. 6 showsthe exemplary syntax.

HEVC further features signaling of restrictions that guarantee a certainsetting for a HEVC coded video sequence, e.g. to indicate a fixed tilestructure in a single layer HEVC coded video sequence (cp.tiles_fixed_structure_flag in VUI syntax given below). Furtherrestrictions of tiles in a scalable coded video sequence are beneficialfor decoder initialization and operation. To allow the decoder to startdecoding the enhancement layer image area that is associated with a baselayer tile after that base layer tile is finished, it is notnecessitated to enforce a complete alignment. Especially in spatialscalability, it might be helpful to allow more tiles in the enhancementlayer than in the base layer. For instance in factor two spatialscalability, four times the number of pels is contained in theenhancement layer image area compared to the corresponding base layerimage area. So it might be helpful to allow four tiles in theenhancement layer for each base layer tile. See FIG. 7, which shows anexample for aligned tile boundaries with spatial scalability. Allvertical boundaries are aligned in the base and enhancement layer. Inthe enhancement layer additional tiles (horizontal boundaries) are usedto allow parallelization using the same number of pels per enhancementlayer tile as in the base layer tile partitioning.

Thus we define tile boundary alignment in a way that only each baselayer boundary has a corresponding boundary in the enhancement layer,but not the other way around. Precisely this means, that for each twoenhancement layer pels, which belong to the same tile, the correspondingbase layer pels also belong to the same tile.

The signaling 606 helps to initialize a parallel decoder environmentwhich, otherwise, would have to gather the information by parsingmultiple parameter sets. Furthermore, in the form of a bitstreamrestriction, the concept of FIG. 4 guarantees that the said restrictionis valid for the complete coded video sequence, for example.

A possible embodiment allows saving signaling of the base layer tileboundaries in the enhancement layer if tile boundaries of the base layerare a subset of the tile boundaries of the enhancement layer.

The information about tile alignment may be signaled explicitly in thebitstream for easy access by the decoder.

In a specific embodiment the signaling could be achieved by using a flagin the VUI parameters of the enhancement layer SPS, as given in FIG. 8:

-   tiles_fixed_structure_flag equal to 1 indicates that each picture    parameter set that is active in the coded video sequence has the    same value of the syntax elements num_tile_columns_minus1,    num_tile_rows_minus1, uniform_spacing_flag, column_width_minus1[i],    row_height_minus1[i] and loop_filter_across_tiles_enabled_flag, when    present. tiles_fixed_structure_flag equal to 0 indicates that tiles    syntax elements in different picture parameter sets may or may not    have the same value. When the tiles_fixed_structure_flag syntax    element is not present, it is inferred to be equal to 0.

Note that the signaling of tiles_fixed_structure_flag equal to 1 is aguarantee to a decoder that each picture in the coded video sequence hasthe same number of tiles distributed in the same way which might beuseful for workload allocation in the case of multi-threaded decoding.

tile_boundaries_aligned_flag corresponds to the structure 606 in FIG. 4.If same is equal to 1 it indicates that all tile boundaries of thecorresponding base layer picture have corresponding tile boundaries inthe given enhancement layer. tile_boundaries_aligned_flag equal to 0indicates that there is no restriction on the tile configuration betweencorresponding base layer and the given enhancement layer.

It is noted that the long-term syntax element structure may guaranteethat, during the predetermined time period, e.g. picture sequence, aminimum number of spatial segments 82 into which the pictures 15 of thesecond layer are subdivided is n times more than the minimum number ofspatial segments 80 into which the pictures 12 of the first layer aresubdivided or that each spatial segment of pictures 12 is made up ofexactly n spatial segments of time-aligned picture 15, with n dependingon the value of the long-term syntax element structure. In the case ofFIG. 7, n would be equal to 3. The decoder would still periodicallydetermine, in time intervals smaller than the predetermined time period,the actual subdivision of the pictures 12, 15 of the first layer and thesecond layer into the spatial segments 80 and 82 based on the short-termsyntax elements of the multi-layer video data stream 40, but again, theguarantee could be exploited by the decoder in order to more efficientlyperform the workload allocation. Another aspect is “opportunisticdecoding”: a decoder with multiple CPU cores may exploit the guaranteeas a hint regarding the parallelism of the layers and to decide thereonto decode layers of higher complexity, i.e. of higher spatial resolutionor higher number of layers. Bitstreams that exceed the capability of asingle core might be decodable by utilizing all cores of the samedecoder. This information is especially helpful, if profile and levelindicators do not involve such indication on minimum parallelism.

A second aspect discussed and presented hereinafter, concerns a conceptwhich could be called “Restricted Inter-Layer Upsampling”: a syntaxelement in the bitstream (c.p., exemplarily,independent_tile_upsampling_idc) is used in order to steer theupsampling filter 36 in case of a spatial scalable multi-layer video. Ifthe upsampling filtering is performed in layer 0 across spatial segmentboundaries 86, then the delay to be met in parallel decoding/encodingthe spatial segments 82 of layer 1 relative to the encoding/decoding ofthe spatial segments 80 of layer 0 is increased as the upsamplingfiltering combines, and thus renders mutually dependent, the informationof neighboring spatial segments of layer 0 to serve as the predictionreference 38 used in inter-layer prediction of blocks 41 of layer 1.See, for example, FIG. 9 Both pictures 12 and 15 are shown in anoverlaying manner with both pictures dimensioned and registered to eachother according to spatial correspondence, i.e. portions showing thesame portion of the scene overlay each other. Pictures 12 and 15 areexemplarily shown to be split into 6 and 12, spatial segments such astiles, respectively. A filter kernel 200 is illustratively shown asmoving across the left-upper tile of picture 12 so as to obtain theupsampled version thereof which serves as a basis for inter-layerpredicting any block within the tiles of picture 15, spatiallyoverlaying the left-upper tile. At some intermediate instances such asat 202 the kernel 200 overlaps a neighboring tile of picture 12. Thesample value of the mid of kernel 200 at position 202 of the upsampledversion thus depends on both samples of the upper-left tile of picture12 as well as samples of the tile of picture 12 to the right thereof. Ifthe upsampled version of picture 12 serves as the basis for inter-layerprediction, the inter-layer offset in parallel processing the segmentsof the layers is increased. A restriction could, thus, help inincreasing the parallelization amount across the different layers and,accordingly, decreasing the overall coding delay. Naturally, the syntaxelement could also be a long-term syntax element which is valid for asequence of pictures. The restriction could be achieved in one of thefollowing ways: filling the overlapping portion of kernel 200 atoverlapping position 202, for example, with a central tendency of thesample values within the non-dashed portion of kernel 200, extrapolatingthe non-dashed portion using linear or other functions into the dashedone or the like.

In order to render the latter aspect even clearer, reference is made toFIG. 10 which shows a decoder 610 receiving a spatially scalablebitstream 40 into which a picture is coded in different spatial layerscorresponding to picture 12 in layer 0 and picture 15 in layer 1. Atleast for one of these spatial layers, decoder 610 is configured todecode same in spatial segments. Coinciding with the description above,these spatial segments may be tiles or substreams or slices. Likewise,the decoder 610 may be configured to subject such spatial segments ofpicture 12 or picture 15 to parallel decoding. That is, base layerpicture 12 may be subdivided into spatial segments such as tiles and/orsubstreams and/or slices and/or enhancement layer picture 15 may besubdivided into tiles and/or substreams and/or slices. As far as thedetails concerning parallel decoding are concerned, reference is made tothe above description of FIG. 4, which may be readily transferred onto adecoder 610 of FIG. 10. That is, in decoding base layer picture 12,decoder 610 uses spatial prediction, and optionally temporal prediction,if base layers 12 and 15 are, for example, part of a layered video. Incase of tiles, spatial prediction is restricted to not cross tileboundaries and the same applies to entropy decoding, which is performedcompletely separately for tiles if entropy decoding is used. The sameapplies to the enhancement layer picture 15, while additionallysupporting inter-layer prediction. As already described above,inter-layer prediction not only concerns prediction parameters of theenhancement layer which are predicted based on corresponding predictionparameters having been used in decoding the base layer, but alsoconcerns predictions derived from reconstructed samples of the baselayer picture at a co-located portion relative to the portion of theenhancement layer picture 15 currently to be predicted using inter-layerprediction. However, as bitstream 40 may be a spatially scalablebitstream, any co-located portion of the base layer picture 12 to formthe basis of inter-layer prediction of a currently processed portion ofthe enhancement layer picture 15 may be to be upsampled by the decoder610 in order to account for the higher spatial resolution of picture 15relative to picture 12. See, for example, FIG. 11. A currently to bepredicted portion of enhancement layer picture 15 is indicated in FIG.11 using reference signs 612. The co-located portion in base layerpicture 12 is indicated using reference sign 614. Owing to the higherspatial resolution of enhancement layer picture 15, the number of samplepositions within portion 612 (indicated by dots) is illustrated to behigher than the samples within portion 614 (likewise indicated by dots).Accordingly, decoder 610 uses interpolation in order to upsample thealready reconstructed version of portion 614 of the base layer picture12. In doing so, the decoder 610 of FIG. 10 is responsive to a syntaxelement 616 in the spatially scalable bitstream 40.

In particular, the just-mentioned responsiveness is explained in moredetail with respect to FIG. 12. FIG. 12 shows portion 614 within baselayer picture 12, i.e. the reference portion in the non-upsampledversion thereof, as well as the corresponding upsampled version thereof,indicated using reference sign 618. As just mentioned, version 618,which is later used for inter-layer prediction by copying, for example,the respective upsampled sample values in portion 618 into the portion612 of enhancement layer 15, is obtained from the base layer picture 12by interpolation 620. However, the interpolation 620 depends on the justmentioned syntax element 616. The way the interpolation 620 changesdepending on syntax element 616 pertains to the areas along bordersbetween neighboring partitions of a partitioning of base layer picture12 and its upsampled version, respectively. In particular, thepartitioning depends on the aforementioned spatial segments into whichat least one of pictures 12 and 15 are subdivided. The partitioning isillustrated in FIG. 11 within the base layer picture 12 using dashedlines 622. The partitioning 622 may, for example, as will be outlined inmore detail below, correspond to a logical AND or logical OR combinationof a spatial overlay of borders of spatial segments of pictures 12 and15 or spatially coincide with the partitioning defined by the spatialsegments of enhancement layer picture 15. In any case, depending onsyntax element 616, decoder 610 either performs the interpolation 620irrespective of the partitioning 622 or considering partitioning 622.When considering the partitioning, decoder 610 performs theinterpolation 620 such that all the samples within upsampled portion 618merely stem, or depend on, or are influenced by, samples of one of thepartitions of partitioning 622, while being independent from any otherpartitions of partitioning 622. If, for example, partitioning 622 iseither a local AND combination of the borders of the spatial segments ofpictures 12 and 15 or a local OR combination thereof, then this meansthat all the samples of interpolated portion 618 stem from merely onepartition of base layer picture 12. If, however, the syntax element 616directs decoder 610 to be insensitive to partitioning 622, it may happenthat different samples within interpolated portion 618 stem fromneighboring partitions of partitioning 622.

612, for example, illustrates the case that a filter kernel 200 is usedin order to perform the interpolation 620 and that, in order to obtainthe interpolated sample encircled in FIG. 12, the kernel 624 overlaps aborder between two neighboring partitions of partitioning 622. In thatcase, the decoder 610 is responsive to syntax element 616 so as toeither fill the filter kernel 624 normally, i.e. by applying the filterkernel 624 onto the respective encompassed samples of base layer picture12 completely, or by filling the fragment of filter kernel 624protruding into the neighboring partition not including portion 614 and618, respectively, indicated using hashing in FIG. 12, using a fallbackrule according to which this respective fragment is filled independentfrom the underlying samples of base layer picture 12. For example, thehashed portion of filter kernel 624 is filled with some mean measure ofthe sample values of the non-hashed portion of filer kernel 624, or byway of some extrapolation. In another alternative, the sample values ofthe base layer picture 12 overlaid by the hashed portion are filledusing a predetermined value such as 0 or the like. Generally, decoder610 could treat the partition border separating the portion encompassingportion 614 and the neighboring partition just like an outer edge ofpicture 12 itself and could use, for example, the same fallback rule inthe interpolation 620 as used in performing the upsampling/interpolationnear or at the outer circumference of picture 12.

In accordance with one embodiment of the present application, thepartitioning 622 is chosen to coincide with the base layer picture'ssubdivision into spatial segments irrespective of any subdivision of theenhancement layer picture 15 into spatial segments. Then, theinter-layer offset between decoding the base layer picture 12 and theenhancement layer picture 15 by decoder 610 may be decreased as portionssuch as portion 614 of the base layer picture 12 do not necessitate thedecoder 610 to have the neighboring partition/spatial segment to bedecoded before performing the inter-layer prediction of portion 612 ofthe enhancement layer picture 15.

Alternatively, decoder 610 could be configured to determine partitioning622 to locally coincide with the subdivision of picture 15 into spatialsegments. In another alternative, decoder 610 could be configured toselect partitioning 622 to be merely made up of borders of spatialsegments of pictures 12 and 15, which spatially coincide, i.e. so as tocorrespond to a logical AND of the borders of pictures 12 and 15—inother words, merely those borders of picture's 15 subdivision intospatial segments form corresponding borders between partitions ofpartitioning 622, which spatially correspond to respective borders ofbase layer picture's 12 subdivision into spatial segments.

It is also feasible, and will be outlined in more detail below, that thesyntax element 616 may direct decoder 610 not only to either disregardpartitioning 622 in interpolating 620 or not, but also distinguishbetween different ways of selecting partitioning 622. See, for example,FIG. 9, where tiles were exemplarily used as spatial segments. If syntaxelement 616 signals to decoder 610 to perform the interpolation 620separately for partitioning 622, decoder 610 could, for example, use theborders of the base layer picture 12 as borders of partitioning 622since they coincide with a finer subdivision of enhancement layerpicture 15 into tiles. Accordingly, in order to commence decoding thesecond tile in the uppermost tile row of enhancement layer picture 15,decoder 610 does not need to wait for the finalization of decoding ofthe second tile in the uppermost row of the base layer picture 12, asthe “interpolation separation” prohibits any mixture of reconstructedsamples of the first two tiles in the uppermost row of the base layerpicture 12. If adopting the enhancement layer picture's 15 subdivisioninto tiles completely, in order to determine partitioning 622, theinterpolation separation is also performed at the dashed lines in FIG. 9and decoder 610 is able to commence decoding the upper left tiles of theenhancement layer picture 15 even earlier, namely as soon as decoder 610has managed to decode the respective co-located sub-portion of the firsttile of the base layer picture 12. In this regard, it is noted that evenin decoding tiles, decoder 610 may use some decoding order which may,for example, involve some raster scan order leading from the respectivetile's upper left corner in a row-wise manner to the right bottom cornerof the respective tile.

That is, in accordance with the second aspect, an encoder formingbitstream 40 is able to choose between two modes by way of syntaxelement 616: if syntax element 616 is set—and inserted into thebitstream 40—to direct decoder 610 to be insensitive to partitioning622, a better inter-layer prediction may be achieved due to the betterinterpolation, but the obtainable degree of parallelization in paralleldecoding pictures 12 and 15 is reduced, i.e. the minimum inter-layeroffset to be obeyed is increased. In the other mode, syntax element 616directs decoder 610 to take the partitioning 622 into account whenperforming interpolation 620 for inter-layer prediction purposes, andaccordingly the inter-layer prediction quality is decreased in favor ofan increased degree of parallelism and a reduced minimum inter-layerdecoding offset in parallel decoding pictures 12 and 15, respectively.

Although the description of the second aspect of the present applicationprimarily focused on a tile subdivisioning or a tile parallel processingconcept, it should be clear that the usage of syntax element 616 inorder to control interpolation 620 is also advantageous when using WPPsubstreams. See, for example, FIG. 13. FIG. 13 illustrates the casewhere the base layer picture 12 is exemplarity subdivided into twosubstreams wherein co-located portions of the enhancement layer picture15 are both subdivided into two substreams each. In case of applying theinterpolation separation responsive to syntax element 616, decoder 610may commence decoding the first, i.e. uppermost, substream ofenhancement layer picture 15, as soon as decoder 610 has decoded thefirst substream of the base layer picture 12 far enough in order tocover the respective left-hand portion of the first enhancement layersubstream of picture 15 and this holds true even for those portions ofthe first enhancement layer substream of picture 15 bordering the secondsubstream of picture 15 as the interpolation separation renders anyinter-layer prediction independent from any reconstructed portions ofthe base layer substream of picture 12 spatially overlaying the secondenhancement layer substream.

Before providing below a detailed embodiment which implements the aboveoutlined switchable restricted inter-layer upsampling, it is noted thatthe syntax element 616 may signal or switch the restriction pertime-frame in case of pictures 12 and 15 being a time-aligned pair ofpictures of a video just as it was the case in FIG. 4, for example.Moreover, again, it is noted that a decoder in accordance with anembodiment of the present application may coincide with the descriptionand functionalities provided above with respect to both FIG. 4 as wellas FIG. 10. In this regard, it is noted that the description providedabove with respect to FIG. 4 concerning the short-term syntax elementsand the signaling of the position and location of the spatial segmentsof pictures 12 and 15 shall be treated as equally applying to theembodiment described with respect to FIGS. 10 to 13. Lastly, It is notedthat the second aspect would also be advantageous if the decoder of FIG.10 was a picture decoder decoding a picture in layers 0 and 1. Thetemporal component is optional.

Implementing restricted inter-layer upsampling in HEVC could be done asfollows. In spatial scalability an upsampled base layer picture is usedto predict the enhancement layer picture. In this process, a predictionvalue is calculated for each pel position of the enhancement layer usingmultiple pel values of the corresponding base layer image area (usuallyin horizontal and vertical direction). If pels from different base layertiles are used, it is not possible to decode an enhancement layer tileonly from base layer tile information covering the same image area asthe enhancement layer tile. Signaling the restricted inter-layerupsampling as a bitstream restriction guarantees a decoder that thespatial partitioning signaled in all parameter sets of a coded videosequence will obey the restriction and thus simplifies initializationand operation of a parallel multi-layer decoder.

The concept of FIG. 10 could be implemented as a mechanism to disallowusage of neighboring pel information for upsampling that are notcontained in the base layer tile that corresponds to the enhancementlayer tile. It is signaled in the bitstream whether the decoder isallowed to use pels outside the corresponding image area of the baselayer at all enhancement layer tile borders.

As an alternative it is signaled in the bitstream whether the decoder isallowed to use pels outside the corresponding image area of the baselayer at all enhancement layer tile borders only for enhancement layertile boundaries that correspond to base layer tile boundaries.

In a specific embodiment the upsampling of base layer at tile boundariesis performed as it would be on a picture boundary where no neighboringpels are available.

In a specific embodiment the signaling could be achieved by using a flagin the picture parameter set of the enhancement layer, as given in FIG.14.

independent_tile_upsampling_idc corresponds to syntax element 612.independent_tile_upsampling_idc not being equal to 0 restricts theupsampling filter to not cross tile boundaries. Ifindependent_tile_upsampling_idc is equal to 2, no base layer samplesthat lie outside the picture area that corresponds to the enhancementlayer tile shall be used for upsampling. Ifindependent_tile_upsampling_idc is equal to 1, the restriction appliesonly to enhancement layer tile boundaries which are aligned with baselayer tile boundaries independent_tile_upsampling_idc equal to 0 doesnot imply this restriction.

Finally, FIG. 15a exemplarily shows two tile-partitioned pictures 12 and15 overlaid in a spatially corresponding manner so as to illustrate thesyntax element's 612 example of FIG. 14: independent_tile_upsampling_idcbeing equal to 2 restricts the upsampling filter to not cross anyenhancement layer tile boundaries. See the dashed-simple-dotted lines400. If independent_tile_upsampling_idc is equal to 1, the restrictionapplies only at enhancement layer tile boundaries which are aligned withbase layer tile boundaries. See the dashed-twice-dotted lines 402independent_tile_upsampling_idc equal to 0 does not imply thisrestriction.

Alternatively to mode independent_tile_upsampling_idc=2, or asadditional mode such as independent_tile_upsampling_idc=3, theupsampling filter could be restricted to not cross any tile boundary,neither those of the base layer nor those of the enhancement layer. Seeline 404 in FIG. 15 b.

That is, depending on the mode, at border 400, 402, or 404, theup-sample filter would be handled as explained above with respect toFIG. 9.

Before turning to the next aspect of the present application, it isbriefly noted that the interpolation 620 discussed above would beperformed, for example, in predictor 60 in FIG. 2 in order to obtain theinter-layer prediction result. As the encoder performs the sameprediction at the encoding side, the interpolation 620 is performeddependent on the setting of the syntax element 616 likewise at theencoder side, such as, for example, within predictor 18. At the encodingside, the decision how to set the syntax element may be performeddepending on the application scenario, for example. In case of low delaybeing of higher importance, for example, the syntax element could be setso as to restrict the inter-layer upsampling, and in other applicationscenarios, it may be more important to have a better prediction and toincrease the compression ratio so that it is more advisable to set thesyntax element 612 so as to not restrict the inter-layer upsampling.

The just-mentioned minimum coding delay—or offset—between the coding ofspatial segments of consecutive layers is the subject of the next aspectas well, which could be named “Layer Decoding Delay Indication”. Thedecoder would be able to determine such minimum decoding delay—oroffset—between coding of spatial segments of picture 15 relative tospatial segments of picture 12 based on short-term syntax elements, butaccording to the next concept, a long-term syntax element structure isused so as to signal this inter-layer—delay—or offset—in advance for apredetermined time period. Again, this helps the decoder in performingthe workload allocation within the parallel decoding of the bitstream40. As a measure for the “delay” or “offset” spatial segments may beused, i.e. the offset may be expressed in units of spatial segments(tiles, slices, or CTB rows for WPP).

In order to describe the latter aspect in more detail, reference is madeto FIG. 16, which largely coincides with FIG. 4. Accordingly, wherefeasible, the same reference signs have been used in FIG. 16 and withrespect to the elements referred to by these common elements, thedescription brought forward above with respect to FIG. 4 shall equallyapply to FIG. 16. It should also be mentioned that the video decoder 640shown in FIG. 16 could, in addition to the functionalities set outbelow, incorporate the functionality described in FIG. 4 with respect tothe long-term syntax element 606. Now, the aspect of the presentapplication of FIG. 16 also uses a long-term syntax element structure,namely 642, which is also inserted into bitstream 40 so as to refer orpertain to the predetermined time period 608. In other words, althoughvideo decoder 640 could be responsive to the syntax element structure606 of FIG. 4 as well as the syntax element structure 642, merely thelatter functionality which is further outlined below is important forthe decoder 640 of FIG. 16, whereas the functionality with respect tothe syntax element structure 606 and the presence of the syntax elementstructure 606 in bitstream 40 is optional for video decoder 640. Thedescription set out above with respect to video decoder 600, however,equally applies to the video decoder 640. That is, the video decoder 640is able to decode the multi-layer video data stream 40, into which ascene is coded in a hierarchy of layers using inter-layer predictionfrom a first layer, layer 0, to a second layer, layer 1. The videodecoder 40 supports parallel decoding the multi-layer video data streamin spatial segments, into which pictures of the layers are partitionedby sequentially traversing the spatial segments in the temporallyoverlapping manner with an inter-layer offset between the traversal ofspatial segments of pictures of the first layer relative to thetraversal of spatial segments of pictures of the second layer. As aminor note it is expressed that spatial segments might be tiles,substreams or slices, but even mixtures of the just mentioned segmentunits are feasible. In combining the slice concept with the tile and/orsubstream concept, the definition of what “spatial segments” actuallyare, may differ.

In any case, the video decoder 640 of FIG. 16 is able to, concerningpictures 12 and 15 of a common time frame 604, decode the spatialsegments of picture 12 on the one hand and spatial segments of picture15 on the other hand in parallel, i.e. in a temporally overlappingmanner. Naturally, to this end the video decoder 640 needs to obey someminimum decoding offset between both layers as due to the inter-layerprediction, the currently decoded portions of the enhancement layer 1within in picture 15 have to belong to the already decoded portion ofthe temporally aligned picture 12 of layer 0.

In case of FIG. 16, the long-term syntax element's structure 642 is usedby the video decoder 640 to determine the inter-layer offset in advanceof the predetermined time period 608.

In connection with the embodiment of FIG. 16, the inter-layer offset isa scalar measure for the “distance” of a first spatial segment ofpicture 12 on the one hand and time-aligned picture 15 on the otherhand. The “distance” is measured spatially. Moreover, in order to bemeaningful, the inter-layer offset, which may be determined based on thelong-term syntax element structure 642, shall be valid for the wholedecoding process of the picture's 12 first spatial segment. That is, allnecessitated reference portions in picture 12 for inter-layer predictionare available for the decoding of the whole first spatial segment ofpicture 15 provided the first “inter-layer offset” spatial segments ofthe base layer picture 12 have previously been decoded.

As described previously, the “currently decoded portion” within picture15 traverses picture 15 in a certain predetermined manner, namely in theaforementioned tile order in case of using tile parallel processing, andin the form of a tilted wavefront in the case of using the WPP conceptusing substreams. The same applies to the spatial segments of the baselayer picture 12. The inter-layer offset determines the portion of thetraversal of picture 12 which has to be already processed before thefirst spatial segment of picture 15 may be subject to decoding at theearliest.

In order to describe this in more detail, reference is made to FIGS. 17aand 17b . FIG. 17a describes the inter-layer offset determinable fromthe long-term syntax element structure 642 in more detail in relation tothe tile concept, while FIG. 17b describes the inter-layer offsetdetermined based on the long-term syntax element structure 642 in moredetail with respect to WPP. Later on, in connection with FIG. 17c , itwill be shown that the inter-layer offset signaling concept using thelong-term syntax element structure 642 is not restricted to the usage ofthe tile and/or WPP concept. Rather, the mere subdivision of a pictureinto slices which are decodable in a self-contained manner by definition(i.e. entropy decoding and spatial intra-picture prediction areperformed completely “in-slice” or independent from neighboring slices),renders the interpretation of an inter-layer offset on the basis of thelong-term syntax element 642 feasible.

FIG. 17a shows two time-aligned pictures 12 and 15, both subdivided intotiles. As became clear from the description of the tile conceptpresented above, normally there is no fixed order within the tiles of apicture 12 or 15 in which they have to be decoded. Rather, tiles may bedecoded in any order. Nevertheless, in connection with the embodiment ofFIG. 16, a tile order is defined at least with respect to the tiles ofbase layer picture 12. The tile order 644 is defined to lead in a rasterscan order row-wise from the top left tile to the bottom right tile ofthe regular arrangement of tiles. In accordance with the embodiment ofFIG. 17a , the inter-layer offset signaled by the long-term syntaxelement structure 642 indicates the number of tiles which have to bealready decoded in tile order 644 of base layer picture 12, in order toallow decoder 640 to commence decoding the first tile of enhancementlayer picture 15. In order to determine the “first tile” among the tilesof enhancement layer picture 15, the first tile of enhancement layerpicture 15 may fixedly be defined as the top left tile of enhancementlayer picture 15. Starting from this first tile of enhancement layerpicture 15, video decoder 640 may adapt a tile order for traversing theenhancement layer tiles of picture 15 depending on the tile subdivisionof picture 12. In the case of FIG. 17a , for example, picture's 12subdivision into tiles comprises two rows and three columns of tiles,whereas picture 15 is subdivided into four rows of tiles with twocolumns of tiles. In that case, it may be favorable for decoder 640 tochoose the tile order for traversing the enhancement layer tiles so asto traverse the left-hand tiles of the first two rows first, thentraversing the right-hand ones of the first two rows, and then repeatingthis traversal with respect to the lower rows of tiles of enhancementlayer picture 15 as shown by arrow 646. In accordance with analternative example which is valid for all aspects described herein,however, the tile decoding order among the enhancement layer tiles ofpicture 15 is fixed and independent from the base layer picturesub-divisioning into tiles. IN cases of using the signaled inter-layercoding offset merely as a trigger for starting/commencing decoding theenhancement layer picture, reorder is not necessitated anyway. Usingdashed lines, FIG. 17a shows the location in picture 12 locallycorresponding to the position of the first tile of enhancement layerpicture 15. As becomes clear from FIG. 17a , in the exemplary case ofFIG. 17a , the inter-layer offset determined by the long-term syntaxelement structure 642 would be “2”, as the first two tiles of picture 12would have had to be decoded before the decoder 640 may commencedecoding the first tile of the enhancement layer picture 15. Merely inthat case are the co-located portions needed for inter-layer predictionavailable in base layer picture 12.

That is, in the case of FIG. 17a , the video decoder 640 would determinefrom the long-term syntax element structure 642 that the inter-layeroffset between the traversal 644 of base layer tiles of picture 12relative to decoding the first tile of the enhancement layer tiles ofpicture 15 is two base layer tiles: the decoding of the first two baselayer tiles in tile order 644 has to be awaited before video decoder 640may commence decoding the first tile of the enhancement layer tiles ofpicture 15 in tile order 646.

FIG. 17b relates to the exemplary case where both time-aligned pictures12 and 15 are subdivided into substreams, namely two substreams in thecase of FIG. 12, and four substreams in the case of FIG. 15. Forexample, substreams may coincide with the aforementioned regularsubdivision of pictures 12 and 15 in rows and columns of coding blocks,namely in a manner so that each substream corresponds to one row of suchcoding blocks. In any case, owing to WPP processing there is, asdescribed above, a decoding order defined among the substreams ofpicture 12, and the substreams of picture 15, respectively, bothdecoding orders 648 and 650 leading from top to bottom. Similar to thecase of FIG. 17a , the decoder 640 is configured to determine from thelong-term syntax element structure 642 the number of leading substreamswhich have to already been decoded before commencing decoding the firstsubstream of picture 15. In the present case, the long-term syntaxelement structure 642 would signal the inter-layer offset to be 1, sincethe complete decoding of the first substream of base layer picture 12suffices in order to provide the necessitated basis for any inter-layerprediction for the first substream of enhancement layer picture 15.

FIG. 17c shows a pair of time-aligned pictures 12 and 15, both beingsubdivided into slices. Again, a slice order or decoding order isdefined among the slices of picture 12 and among the slices of picture15, respectively, both orders 652 and 654 leading from top to bottom. Inthe exemplary case of FIG. 17c , the borders between the slices withinpictures 12 on the one hand and picture 15 on the other hand, locallycorrespond to each other. Accordingly, depending on the “blurring”introduced by the inter-layer prediction from base layer picture 12 toenhancement layer picture 15, the long-term syntax element structure 642will either signal the inter-layer offset to be equal to 1 or equal to2. In particular, owing to enlargements of corresponding co-locatedreference portions in picture 12 for inter-layer predicted portions ofpicture 15, due to, for example disparity compensation vector or due toupsampling interpolation filter kernels as discussed above with respectto FIG. 9, for example, the first two slices of picture 12 in sliceorder 652 may have to be already decoded before the decoder 640 is ableto commence decoding the first slice of enhancement layer picture 15.If, however, such blurring options of inter-layer prediction are eitherswitched off or do not apply because of, for example, the samplingresolution between pictures 12 and 15 being equal to each other and bothpictures 12 and 15 pertaining to the same view so that no disparitycompensation takes place, then the long-term syntax element structurewould be set to be equal to 1 by the encoder so that the decoder 640 maycommence decoding the first slice of enhancement layer picture 15 assoon as the first slice of base layer picture 12 has been completelydecoded.

Accordingly, the description of FIGS. 16 to 17 c has revealed that theusage of long-term syntax element structure 642 helps the encoder tosupport the decoder in scheduling any parallel decoding of time alignedpictures 12 and 15, namely by informing the decoder on the basis of thelong-term syntax element's structure about the inter-layer offset whichis valid for the whole predetermined time period 608 and pertains to thenumber of spatial segments of the base layer picture 12 which has to bealready decoded before commencing the first spatial segment of thetime-aligned picture 15. It should be noted that the video decoder 640would be able to determine the inter-layer offset signaled by way of thelong-term syntax element structure 642 even by itself based on aninspection/evaluation of the short-term syntax elements 602 and furthersyntax elements concerning potential options relating to the inter-layerprediction with these options switching on or off the just-outlinedblurring in inter-layer predicting from base layer to enhancement layer.However, video decoder 640 will have to inspect a multitude of syntaxelements in order to derive the same information as provided by thelong-term syntax element structure 642, and video decoder 640 wouldmerely be able to derive the same information on a short-term basisrather than in advance for the long predetermined time period 608.

Similar to aspects 1 and 2, in the following, a possible way ofintroducing the delay indication aspect into HEVC is describedhereinafter.

First of all, with respect to FIG. 18, it is described how WPP iscurrently implemented in HEVC. That is, this description shall also forma basis for optional implementations of the WPP processing of any of theabove embodiments.

In the base layer, wavefront parallel processing allows parallelprocessing of coded tree block (CTBs) rows. Prediction dependencies arenot broken across CTB rows. With regards to entropy coding, WPP changesthe CABAC dependencies to the top-left CTB in the respective upper CTBrow, as can be seen in FIG. 18. Entropy coding a CTB in following rowscan start once entropy decoding of the corresponding upper-right CTB isfinished.

In the enhancement layer, decoding of a CTB can start as soon as theCTBs containing the corresponding image area are fully decoded andavailable.

The decoding delay or offset indication concept of FIG. 16 may end-up ina signaling that facilitates initialization and operation of a decoderfor layered bitstreams that make use of Tiles, WPP or Slices forparallelization.

Decoding of enhancement layer CTBs when using spatial scalability canstart only once the base layer CTBs covering the corresponding imagearea are available. When parallelizing WPP with a layered bitstream withspatial scalability, layers may differ in terms of picture size, e.g.picture size scaling factors of 1.5 and 2 between layers are stated inthe issued calls for proposals for a scalable extension of HEVC [1], andadditionally maximum CTB sizes, 16, 32 and 64 image samples aresupported in the main profile of HEVC. For quality scalability, thepicture size scaling factor is usually constant but the maximum CTBsizes may still differ between layers.

The ratio between maximum CTB size and picture size scaling factors ofthe layers influence the layer decoding delay, i.e. the CTB row offsetbefore first CTB row of the enhancement layer can be decoded withrespect to the decoding of the base layer CTB rows. FIG. 19 reports theratio of CTB that cover corresponding image areas in both layers withexemplary parameter values for picture size scaling factor and CTB sizesand the introduced layer decoding delay in terms of CTB rows.

For quality scalability between layers, the picture size scaling factorbetween layers is usually equal to 1 while the maximum CTB sizes in therespective layers may still differ and influence the layer decodingdelay.

Syntax element structure 642 provides a decoder hint in the bitstreamthat signals the layer decoding delay for dependent spatial or qualityenhancement layers when WPP processing is parallelized across spatialenhancement layers.

An embodiment uses picture size scaling factors and maximum CTB sizescaling factors between the respective layers to determine the signaledlayer decoding delay.

Depending on the type of scalability between independent base anddependent enhancement layer bitstream, the factors that influence thelayer decoding delay differ.

In multiview scalability, layers represent camera views and mechanismsof inter layer prediction are used to perform prediction between cameraview from various angles. The prediction uses mechanisms of motioncompensation to compensate the differing position of cameras within thecamera setup. In this case, the layer decoding delay is furtherrestricted by the maximum or actual motion vectors in vertical directioncompared to the case of spatial or quality scalability.

Syntax element structure 642 describes a decoder hint in the bitstreamthat signals the layer decoding delay for dependent camera views whenWPP processing is parallelized across multiple camera views.

An embodiment uses picture size scaling factors, maximum CTB sizescaling factors and maximum motion vector length in vertical directionbetween the respective layers to determine the signaled layer decodingdelay.

An embodiment signals the layer decoding delay for spatial, quality ormultiview scalability in terms of spatial segments, i.e. rows of CTBswhen WPP is used, in the VUI syntax of the SPS associated with theenhancement layer or in a compiled for within the VPS extension syntax.

Tiles and Slices

Parallel processing with other partitioning techniques like tiles orslices can also benefit from a hint within the bitstream that indicatesa decoding delay based on the division of pictures into spatialsegments, i.e. tiles or slices. Information of the base layer (e.g.reconstructed image data) may be necessitated for the enhancement layerdecoding process.

Syntax element structure 642 describes a decoder hint in the bitstreamthat signals the layer decoding delay for tiles and/or slices.

A possible embodiment of the invention uses spatial segments as unit toexpress an introduced layer processing delay depending on the type ofparallelization technique used in the coded video sequence.

The syntax in FIG. 20 gives an exemplary embodiment for indication ofmin_spatial_segments_delay (an example for syntax element structure 642)in VUI parameters of the enhancement layer SPS for the parallelizationtools WPP, tiles and slices.

min_spatial_segment_delay describes the decoding delay of the currentlayer introduced by coding dependencies with respect to thecorresponding base layer in terms of spatial segments.

Depending on the value of min_spatial_segment_delay, the followingapplies:

-   If min_spatial_segment_delay is equal to 0, no limit on the minimum    delay between decoding of layers is signaled

Otherwise (min_spatial_segment_delay is not equal to 0), it is arequirement of bitstream conformance that exactly one of the followingconditions shall be true:

-   -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 0 and        entropy_coding sync_enabled_flag is equal to 0 (i.e. neither        tiles nor WPP are used in the video sequence), and all base        layer resources for the decoding process of first slice of the        current layer in bitstream order are available when the first        min_spatial_segment_delay slices of the base layer in bitstream        order are decoded completely.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 1 and        entropy_coding sync_enabled_flag is equal to 0 (i.e. tiles are        used in the video sequence), and all base layer resources for        the decoding process of the first tile of the current layer in        bitstream order are available when the first        min_spatial_segment_delay tiles are completely decoded that        cover the same image area.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 0 and        entropy_coding sync_enabled_flag is equal to 1 (i.e. WPP is used        in the coded video sequence), and all base layer resources for        the decoding process of the first CTB row in the current layer        are available when the first min_spatial_segment_delay CTB rows        of the base layer are completed.

Another exemplary embodiment is the indication ofmin_spatial_segments_delay in the VPS extension syntax as reported in[4] and extended as shown in FIG. 21.

min_spatial_segment_delay describes the decoding delay of the layer[i]introduced by coding dependencies with respect to the corresponding baselayer in terms of spatial segments.

Depending on the value of min_spatial_segment_delay, the followingapplies:

-   If min_spatial_segment_delay is equal to 0, no limit on the minimum    delay between decoding of layers is signaled

Otherwise (min_spatial_segment_delay is not equal to 0), it is arequirement of bitstream conformance that exactly one of the followingconditions shall be true:

-   -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to O and        entropy_coding sync_enabled_flag is equal to O (i.e. neither        tiles nor WPP are used in the video sequence), and all base        layer resources for the decoding process of first slice of the        current layer in bitstream order are available when the first        min_spatial_segment_delay slices of the base layer in bitstream        order are decoded completely.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 1 and        entropy_coding sync_enabled_flag is equal to 0 (i.e. tiles are        used in the video sequence), and all base layer resources for        the decoding process of the first tile of the current layer in        bitstream order are available when the first        min_spatial_segment_delay tiles are completely decoded that        cover the same image area.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 0 and        entropy_coding sync_enabled_flag is equal to 1 (i.e. WPP is used        in the coded video sequence), and all base layer resources for        the decoding process of the first CTB row in the current layer        are available when the first min_spatial_segment_delay CTB rows        of the base layer are completed.

Various prediction modes supported by encoder and decoder as well asrestrictions imposed onto prediction modes as well as context derivationfor entropy coding/decoding in order to enable the parallel processingconcepts, such as the tile and/or WPP concept, have been describedabove. It has also been mentioned above that encoder and decoder mayoperate on a block basis. For example, the above explained predictionmodes are selected on a block basis, i.e. at a granularity finer thanthe pictures themselves. Before proceeding with describing anotheraspect of the present application, the relation between slices, tiles,WPP substreams and the just mentioned blocks shall be explained.

FIG. 32 shows a picture which may be a picture of layer 0, such as layer12 or a picture of layer 1 such as picture 15. The picture is regularlysubdivided into an array of blocks 90. Sometimes, these blocks 90 arecalled largest coding blocks (LCB), largest coding units (LCU), codingtree blocks (CTB) or the like. The subdivision of the picture intoblocks 90 may form a kind of base or coarsest granularity at which theabove described predictions and residual codings are performed and thiscoarsest granularity, i.e. the size of blocks 90, may be signaled andset by the encoder, individually for layer 0 and layer 1. For example, amulti-tree such as a quad-tree subdivision may be used and signaledwithin the data stream so as to subdivide each block 90 into predictionblocks, residual blocks and/or coding blocks, respectively. Inparticular, coding blocks may be the leaf blocks of a recursivemulti-tree subdivisioning of blocks 90 and some prediction relateddecisions may be signaled at the granularity of coding blocks, such asprediction modes, and the prediction blocks at the granularity of whichthe prediction parameters such as motion vectors in case of temporalinter prediction and disparity vectors in case of inter layer predictionfor example, is coded and residual blocks at the granularity of whichthe prediction residual is coded, may be the leaf blocks of separaterecursive multi-tree subdivisionings of the code blocks.

A raster scan coding/decoding order 92 may be defined among blocks 90.The coding/decoding order 92 restricts the availability of neighboringportions for the purpose of spatial prediction: merely portions of thepicture which according to the coding/decoding order 92 precede thecurrent portion such as block 90 or some smaller block thereof, to whicha currently to be predicted syntax element relates, are available forspatial prediction within the current picture. Within each layer, thecoding/decoding order 92 traverses all blocks 90 of the picture so as tothen proceed with traversing blocks of a next picture of the respectivelayer in a picture coding/decoding order which not necessarily followsthe temporal reproduction order of the pictures. Within the individualblocks 90, the coding/decoding order 92 is refined into a scan among thesmaller blocks, such as the coding blocks.

In relation to the just outlined blocks 90 and the smaller blocks, eachpicture is further subdivided into one or more slices along the justmentioned coding/decoding order 92. Slices 94 a and 94 b exemplarilyshown in FIG. 32 accordingly cover the respective picture gaplessly. Theborder or interface 96 between consecutive slices 94 a and 94 b of onepicture may or may not be aligned with borders of neighboring blocks 90.To be more precise, and illustrated at the right hand side of FIG. 32,consecutive slices 94 a and 94 b within one picture may border eachother at borders of smaller blocks such as coding blocks, i.e. leafblocks of a subdivision of one of blocks 90.

Slices 94 a and 94 b of a picture may form the smallest units in whichthe portion of the data stream into which the picture is coded may bepacketized into packets, i.e. NAL units. A further possible property ofslices, namely the restriction onto slices with regards to, for example,prediction and entropy context determination across slice boundaries,was described above. Slices with such restrictions may be called“normal” slices. As outlined in more detail below, besides normal slices“dependent slices” may exist as well.

The coding/decoding order 92 defined among the array of blocks 90 maychange if the tile partitioning concept is used for the picture. This isshown in FIG. 33 where the picture is exemplarily shown to thepartitioned into four tiles 82 a to 82 d. As illustrated in FIG. 33,tiles are themselves defined as a regular subdivision of a picture inunits of blocks 90. That is, each tile 82 a to 82 d is composed of anarray of n×m blocks 90 with n being set individually for each row oftiles and m being individually set for each column of tiles. Followingthe coding/decoding order 92, blocks 90 in a first tile are scanned inraster scan order first before proceeding to the next tile 82 b and soforth, wherein the tiles 82 a to 82 d are themselves scanned in a rasterscan order.

In accordance with a WPP stream partitioning concept, a picture is,along the coding/decoding order 92, subdivided in units of one or morerows of block 90 into WPP substreams 98 a to 98 d. Each WPP substreammay, for example, cover one complete row of blocks 90 as illustrated inFIG. 34.

The tile concept and the WPP substream concept may, however, also bemixed. In that case, each WPP substream covers, for example one row ofblocks 90 within each tile.

Even the slice partitioning of a picture may be co-used with the tilepartitioning and/or WPP substream partitioning. In relation to tiles,each of the one or more slices the picture is subdivided into may eitherbe exactly composed of one complete tile or more than one complete tile,or a sub-portion of merely one tile along the coding/decoding order 92.Slices may also be used in order to form the WPP substreams 98 a to 98d. To this end, slices forming the smallest units for packetization maycomprise normal slices on the one hand and dependent slices on the otherhand: while normal slices impose the above-described restrictions ontoprediction and entropy context derivation, dependent slices do notimpose such restrictions. Dependent slices which start at the border ofthe picture from which the coding/decoding order 92 substantially pointsaway row-wise, adopt the entropy context as resulting from entropydecoding block 90 in the immediately preceding row of blocks 90, anddependent slices starting somewhere else may adopt the entropy codingcontext as resulting from entropy coding/decoding the immediatelypreceding slice up to its end. By this measure, each WPP substream 98 ato 98 d may be composed of one or more dependent slices.

That is, the coding/decoding order 92 defined among blocks 90 linearlyleads from a first side of the respective picture, here exemplarily theleft side, to the opposite side, exemplarily the right side, and thensteps to the next row of blocks 90 in downward/bottom direction.Available, i.e. already coded/decoded portions of the current picture,accordingly lie primarily to the left and to the top of the currentlycoded/decoded portion, such as the current block 90. Due to thedisruption of predictions and entropy context derivations across tileboundaries, the tiles of one picture may be processed in parallel.Coding/decoding of tiles of one picture may even be commencedconcurrently. Restrictions stem from the in-loop filtering mentionedabove in case where same is allowed to cross tile boundaries. Commencingthe coding/decoding of WPP substreams, in turn, is performed in astaggered manner from top to bottom. The intra-picture delay betweenconsecutive WPP substreams is, measured in blocks 90, two blocks 90.

However, it would be favorable to even parallelize the coding/decodingof pictures 12 and 15, i.e. the time instant of different layers.Obviously, coding/decoding the picture 15 of the dependent layer has tobe delayed relative to the coding/decoding of the base layer so as toguarantee that there are “spatially corresponding” portions of the baselayer already available. These thoughts are valid even in case of notusing any parallelization of coding/decoding within any of pictures 12and 15 individually. Even in case of using one slice in order to coverthe whole picture 12 and 15, respectively, with using no tile and no WPPsubstream processing, coding/decoding of pictures 12 and 15 may beparallelized. The signaling described next, i.e. aspect six, is apossibility to express such decoding/coding delay between layers even insuch a case where, or irrespective of whether, tile or WPP processing isused for any of the pictures of the layers.

As became clear from the above description relating to the minimumcoding delay between the coding of consecutive layers it became clearthat the decoder would be able to determine the minimum decoding delaybased on short-term syntax elements. However, in case of using long-termsyntax elements as described above, so as to signal this inter-layertemporal delay in advance for a predetermined time period, the decodermay plan into the future using the guarantee provided and may moreeasily perform the workload allocation within the parallel decoding ofthe bitstream 40.

The aspect of the present application described next, namely the sixthaspect of the present application, is somehow related to aspect 3 inthat the explicit signaling of any inter-layer offset is concerned.However, with respect to the sixth aspect of the present application,the syntax element structure signaling the inter-layer offset explicitlydoes not need to be signaled on a long-term basis relative to theshort-term syntax element from which the inter-layer offset wouldotherwise be derivable. Rather, the sixth aspect of the presentapplication exploits another finding: as became clear when describingFIGS. 17a to 17c , in case of base and enhancement layer pictures bothbeing subdivided into blocks with a raster scan decoding order beingdefined thereamong, then the inter-layer offset between base andenhancement layer may efficiently be explicitly signaled by a measurewhich measures the inter-layer offset in units of base layer blocks. Inconnection with the aspect described further below, the base layerblocks in units of which the inter-layer offset is explicitly signaledis not restricted to spatial segments. Rather, other coding blocks maybe used to this end. Accordingly, when describing an embodimentconcerning the sixth aspect of the present application with respect toFIG. 34, the reference signs having been used with respect to FIGS. 16to 17 c are largely co-used and the description brought forward abovewith respect to the latter figures shall equally apply to the embodimentdescribed further below in order to avoid unnecessitated repetition.Beyond this, reference is made to the description of FIGS. 32 and 33, asthese figures form a possible coexistence between coding blocks on theone hand and spatial segments on the other hand.

Thus, FIG. 35 shows a video decoder 720 configured to receive amulti-layer video data stream 40 into which a scene is coded in ahierarchy of layers using inter-layer prediction from portions of afirst layer to co-located portions of a second layer with, similar tothe above figures, exemplarily showing layer 0 in juxtaposition to layer1 as a representative example. Two-time aligned pictures 12 and 15 ofboth layers are exemplarily shown in FIG. 35. The pictures 12 of thebase layer 0 are subdivided into an array of first blocks 722 and thepictures of the enhancement layer 1 are subdivided into an array ofsecond blocks 724. A raster scan decoding order 726 is defined amongblocks 722, and likewise a raster scan decoding order 728 is definedamong blocks 724.

The video data stream 40 comprises a syntax element structure whichindicates an inter-layer offset for parallely decoding pictures 12 and15 by sequentially traversing the first and second blocks 722 and 724 ina temporally overlapping manner with the inter-layer offset between thetraversal of the first and second blocks 722 and 724, measured in unitsof the base layer blocks 722. The video decoder 720 is configured to beresponsive to this syntax element structure 730. In particular, thevideo decoder determines the inter-layer offset from the latter.

In agreement with the embodiment of FIG. 16, the syntax elementstructure 730 may indicate the inter-layer offset as a guarantee forsuccess for the performing parallel decoding of time-aligned pictures 12and 15 for a predetermined time period longer than the short-term timeintervals at which optional syntax elements signal pictures' 12 and 15subdivision into blocks 722 and 724, respectively. This is, however, notmandatory. Rather, the explicit signaling of the inter-layer offset byway of the syntax element structure 730 may be realized in data stream40 at different scope such as for each time-aligned pair of base andenhancement layer pictures 12 and 15, for example, i.e. at the sameinterval as the signalization concerning the size and subdivision intoblocks 722 and 724.

As a further agreement with the above embodiment, decoder 720 may usethe inter-layer offset explicitly signaled by way of the syntax elementstructure 730 as a measure for the offset in commencing the traversal ofthe second blocks 724 relative to the traversal of the first blocks 722in parallely decoding pictures 12 and 15, respectively. In other words,video decoder 720 may be configured to derive a count value from thesyntax element structure 730, count the already decoded blocks 722 ofbase layer picture 12 along raster scan decoding order 726 whileallowing commencing the decoding of the sequence of blocks 724 of theenhancement layer picture 15 along the decoding order 728 merely afterthe count of already decoded blocks 722 reaching the minimum countexplicitly signaled by the syntax element structure 730. Accordingly,video decoder 720 does not need to inspect any highly complicated anddistributed portions of the video data stream 40, which would otherwiseenable video decoder 720 to otherwise compute an actual minimuminter-layer offset between commencing decoding blocks 722 on the onehand and blocks 724 on the other hand.

Interestingly, however, according to the embodiment of FIG. 35, blocks722 and 724 do not necessarily represent spatial segments beingespecially dedicated for being subject to any parallel processing.Rather, blocks 722 and 724 may be usual coding blocks in units of whichthe content of pictures 12 and 15 is coded into video data stream 40,respectively. For example, blocks 722 and 724 may be tree root blocksinto which pictures 12 and 15 are regularly, i.e. in rows and columns,subdivided, which tree root blocks are then further, individually,subdivided in a recursive multi-tree manner as described above withrespect to FIG. 32, for example, with the resulting leaf blocks of thetree root blocks representing the pictures' 12 and 15 subdivision intocoding blocks in units of which the prediction mode—selected amongspatial, temporal and, with respect to FIG. 15, inter-layerprediction—is signaled in the video data stream 40.

In order to explain the embodiment of FIG. 35 in more detail, referenceis made to FIG. 36. As shown in FIG. 36, the video decoder 720 may use acounter 732 in order to count the number of already decoded blocks 722of base layer picture 12, with starting the count from the first block722 of picture 12 in raster scan decoding order 726. A comparator 734 ofdecoder 720 compares this steadily increasing count output by counter732, with the explicitly signaled value of the syntax element structure730 as obtained from the video data stream 40. In case of the countfulfilling a predetermined relationship to the value indicated by thesyntax element structure 730, such as, as soon as the count of counter732 reaches or equals the value indicated by the syntax elementstructure 730, comparator activates or causes the commencing of decodingof enhancement layer picture 15, i.e. causes the decoding of the firstblock 724 of the enhancement layer picture 15 in raster scan decodingorder 728 to be started.

As will be described in more detail below with respect to explicitexamples for a syntax for a syntax element structure 730, it would befeasible to unify the syntax element structure of FIG. 35 with thesyntax element structure and its aim of FIG. 16. This means thefollowing. The syntax element structure 730 could have a set of possiblevalues, i.e. a possible-values-set. A value out of anon-explicit-inter-layer-offset set of possible values could cause thevideo decoder 720 to disregard the value of the syntax element structure730 and either not perform parallel decoding pictures 12 and 15 ordetermine any inter-layer offset on the basis of the short term syntaxelements 602 as exemplarily mentioned with respect to FIG. 16. In caseof the syntax element structure 730 assuming a value out of a second setof possible values, this would cause the video decoder 720 to performthe action already outlined with respect to FIG. 36, for example,according to which the value of the syntax element structure 730 wouldexplicitly signal the inter-layer offset in units of the base layerblocks 722. However, there could be another subset of the possiblevalues of the syntax element structure 730, which, when being assumed bythe syntax element structure 730, causes video decoder 720 to act asdescribed above with respect to FIG. 16: determine the inter-layeroffset between base layer picture's 12 and enhancement layer picture's15 decoding based on the syntax element structure's 730 value byinterpreting the latter as measuring the inter-layer offset in units ofspatial segments which may, but do not have to be, composed of integerones of blocks 722 and 724, respectively.

The just mentioned possibility of combining the embodiment of FIG. 35with that of FIG. 16 is illustrated with respect to FIG. 37. As shown inFIG. 37, the video decoder may inspect the syntax element structure 730in order to determine whether same has a value of a first subset 736, asecond subset 738, or a third subset 740 out of a set of possible values742. Depending on the investigation or check outcome, the decoder 720either derives no guarantee out of the syntax element structure 730 anddoes not derive any explicit signaling concerning the inter-layer offsetfrom the syntax element structure 730 or performs such deriving of theinter-layer offset from the syntax element structure 730, namely eitherin units of spatial segments or in units of blocks. No deriving/noguarantee takes place in case of the second subset 738, the deriving ofthe inter-layer offset in units of spatial segments takes place in caseof subset 736, and deriving the inter-layer offset in units of blockstakes place in case of the syntax element 730 assuming a value out ofthe third subset 740. In the specific syntax example outlined furtherbelow, the syntax element structure comprises two flags, namelyctb_delay_enabled_flag and min_spatial_segment_delay, wherein ctp delayenabled flag=0 and min_spatial_segment_delay ≠0 corresponds to the caseof subset 736, min_spatial_segment_delay=0 corresponds to the secondsubset 738, and ctp_delay_enabled_flag=1 and min_spatial_segment_delay≠0 correspond to the third subset 740.

Finally, with respect to FIG. 38, it is shown that decoder 720 may beconfigured to interpret any inter-layer offset signaled by the syntaxelement structure 730, to be not only an inter-layer offset with respectto the commencing of the decoding of the first block or spatial segmentof the enhancement layer picture 15, but a continuous inter-layer offsetwhich, when being obeyed, results in a conflict-free parallel decodingof pictures 12 and 15, respectively. As shown in FIG. 38, the counter732 still counts the number of already decoded blocks 722 of the baselayer picture 12, wherein, however, an additional counter 744 likewisecounts the already decoded blocks 724 of the enhancement layer picture15 along decoding order 728 wherein a subtractor 746 forms a differenceout of both counts, namely s and t−1, i.e. computes s−t+1. Thisdifference value is compared by comparator 734 with the inter-layeroffset value derived from the syntax element structure 730 and as soonas both values, i.e. the derived inter-layer offset and the differencevalue of the counts have a predetermined relationship, such as thedifference value being equal or exceeding the derived inter-layeroffset, the decoding of block t in decoding order 728 among theenhancement layer blocks 724 is commenced. By this measure, a continuoussurvey between the cause of decoding blocks 722 of the base layerpicture 12 on the one hand and blocks 724 of the enhancement layerpicture 15 on the other hand is installed.

It is clear that the continuous survey according to FIG. 38 may also beapplied to spatial segments. More generally, the description of FIGS. 38and 36 may be readily transferred onto spatial segments as well and thisstatement also applies to the embodiment of FIG. 16 where the syntaxelement structure 642 may serve as the relevant syntax element structureshown at 730 in FIGS. 36 and 38. In other words, at least when usingtiles as the spatial segments, there is also a raster scan decodingorder defined thereamong so that the concept of FIG. 36 and FIG. 38discussed with respect to coding blocks may be easily transferred ontothe traversal of tiles and their decoding.

Briefly summarizing the sixth aspect and the embodiments described withrespect thereto, the syntax element structure 730 may be inserted intothe bitstream by a video encoder so as to provide the decoder with anexplicit hint how to control the parallel decoding of base andenhancement layer pictures relative to each other. The inter-layeroffset explicitly signaled by way of the syntax element structure may beactivated or deactivated. If activated, the indication may be in unitsof blocks such as CTBs or, by way of more precise signaling, in asignaled one of units of such blocks and units of spatial segments. Dueto the use of a raster scan order among base layer blocks on the onehand and enhancement layer blocks on the other hand, both leading, forexample, row-wise from the left top corner to the right bottom corner ofeach picture 12/15 row by row from top to bottom, the inter-layer offsetexplicitly signaled by either be interpreted solely as a “trigger” forstarting/commencing decoding the first block of the enhancement layerpicture 15, or as a continuous “safety distance” between the currentlydecoded block of the base layer picture 12 on the one hand and thecurrently decoded block of the enhancement layer picture 15 on the otherhand, i.e. for determining triggers for the decoding of each block ofthe enhancement layer picture 15. The description brought forward withrespect to the sixth aspect may be transferred onto the description andthe embodiments of the third aspect in that at least as far as thedescription of tiles as spatial segments is concerned, theinterpretation and check of abeyance of the signaled inter-layer offsetis concerned, the embodiments of FIGS. 36 and 38 may be used, thetraversal of the decoded tiles within base and enhancement layerpictures along the raster scan tile decoding order may be controlled ina manner corresponding to the description of FIGS. 36 and 38.

Thus, as a measure for the “delay” spatial segments may be used, i.e.the delay may be expressed in units of spatial segments (tiles, slices,or CTB rows for WPP), or the delay/offset may be measured in units ofblocks 90.

The High Efficiency Video Coding (HEVC) standard may be extended toconform to the sixth aspect as follows. In particular, parallel decodingof individual layers (or views) is allowed, if reference data isavailable. The minimum delay (specifically the layer decoding delay)between the decoding of the base layer coded tree block (CTB) and thedecoding of the corresponding dependent enhancement layer CTB isdetermined through the granularity of parallelization tools such astiles, wavefront or slice or the motion compensation vector isapplicable (e.g. in stereoscopic or multiview video coding).

FIG. 20 showed a layer decoding delay indication realized by enhancementlayer Sequence Parameter Set Syntax and semantics by the syntax elementmin_spatial_segment_delay.

min_spatial_segment_delay described the decoding delay of the currentlayer introduced by coding dependencies with respect to thecorresponding base layer in terms of spatial segments.

A mechanism in the HEVC high level syntax that allows optionallyexpressing the layer decoding delay between an independent relative baselayer and a dependent enhancement layer regardless of potentialparallelization techniques in use based on a number of vertical andhorizontal CTBs may be achieved as follows.

A flag or index (e.g. ctb_delay_enabled_flag) may be used to signal thatthe layer decoding delay (as signaled with a second syntax element) isexpressed as a specific CTB address in the coded picture.

A CTB address in raster scan order clearly defines a horizontal avertical position within the picture that is used to express the delayas can be seen from FIG. 39 and according to the following.

CTB coordinates=(CTB address % PicWidthInCTBs, CTBaddress/PicWidthInCTBs)

PicWidthInCTBs describes the width of the pictures in units of CTBs.

FIG. 39 shows the following. A CTB address, e.g. 7, within a picturedefines a horizontal CTB column and vertical CTB row, e.g. the tuple(2,1).

If the flag is enabled, when decoding a CTB in the current dependentlayer, the value of another syntax element (cp.min_spatial_segment_delay) is to be interpreted as an offset to a CTBaddress of a co-located CTB in the relative base layer picture.

The co-located CTB can be calculated according to the sizes of CTBs inthe two respective layers and the widths of the pictures of the tworespective layers, as illustrated in FIG. 40 and detailed in the courseof the following description.

FIG. 40 contains three examples from left to right that show varioussetups of CTB sizes and pictures sizes in two respective base andenhancement layers, irrespective of picture scaling. The bold lined boxin the base layer pictures marks an image area in the size of anenhancement layer CTB and its co-located image area in the CTB layout ofthe respective base layer.

An embodiment of such an optional CTB based layer decoding delayindication is given in the enhancement layer Sequence Parameter SetSyntax and semantics by the syntax element min_spatial_segment_delay inFIG. 41.

ctb_based_delay_enabled_flag being equal to 1 indicates that the delaysignaled with min_spatial_segment_delay is given in units of CTBs.ctb_based_delay_enabled_flag indicates that min_spatial_segment_delay isnot given in units of CTBs.

min_spatial_segment_delay describes the decoding delay of the currentlayer introduced by coding dependencies with respect to thecorresponding base layer in terms of spatial segments.

Depending on the value of min_spatial_segment_delay, the followingapplies:

-   If min_spatial_segment_delay is equal to 0, no limit on the minimum    delay between decoding of layers is signaled

Otherwise (min_spatial_segment_delay is not equal to 0) and ifctb_based_delay_enabled_flag is equal to 1, it is a requirement of bitstream conformance that the following condition shall be true:

-   -   With CtbSizeY_(A), PicWidthInCtbsY_(A) and ctbAddrRsA being the        CtbSizeY and PicWidthInCtbsY of a base layer A and the Ctb        Address of a Ctb in base layer A in raster scan order,        CtbSizeY_(B), PicWidthInCtbsY_(B) and ctbAddrRs_(B) being the        CtbSizeY and PicWidthInCtbsY of a dependent layer/view Band the        Ctb Address of a Ctb of a dependent layer B in raster scan order        and CtbScalingFactors_(BA), CtbRow_(BA)(ctbAddrRs) and        CtbCol_(BA)(ctbAddrRs) being determined as follows

-   Ctb ScalingFactorsu_(A)=(PicWidthInCtbsY_(A)/PicWidthInCtbsY_(B))

-   CtbRows_(BA)(ctbAddrRs)=

-   Ceil((Floor(ctbAddrRs/PicWidthInCtbsY_(B))+1) *    CtbScalingFactor_(BA))−1

-   CtbCols_(BA)(ctbAddrRs)=

-   Ceil(((ctbAddrRs % PicWidthInCtbsY_(B))+1) *    CtbScalingFactor_(BA))−1

When decoding the CTB with ctbAddrRss of the current enhancementlayer/view B, all necessitated base layer resources are available whenthe base layer CTB with ctbAddrRs_(A) being equal toPicWidthInCtbsY_(A) *CtbRow_(BA)(ctbAddrRs_(B))+CtbCol_(BA)(ctbAddrRs_(B))+min_spatial_segment_delayis decoded completely.

Otherwise (min_spatial_segment_delay is not equal to 0 andctb_based_delay_enabled is equal to 0), it is a requirement of bitstream conformance that exactly one of the following conditions shall betrue:

-   -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 0 and        entropy_coding sync_enabled_flag is equal to 0 (i.e. neither        tiles nor WPP are used in the video sequence), and all base        layer resources for the decoding process of first slice of the        current layer in bit stream order are available when the first        min_spatial_segment_delay slices of the base layer in bitstream        order are decoded completely.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 1 and        entropy_coding sync_enabled_flag is equal to 0 (i.e. tiles are        used in the video sequence), and all base layer resources for        the decoding process of the first tile of the current layer in        bit stream order are available when the first        min_spatial_segment_delay tiles are completely decoded that        cover the same image area.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 0 and        entropy_coding sync_enabled_flag is equal to 1 (i.e. WPP is used        in the coded video sequence), and all base layer resources for        the decoding process of the first CTB row in the current layer        are available when the first min_spatial_segment_delay CTB rows        of the base layer are completed.

Alternatively, the inter-layer offset may be signaled as a kind of worstcase delay for ctb_based_delay_enabled_flag instead of a start-up delayfor the first slices/tile/CTB row as in the former embodiments. A worstcase delay gives a guarantee that during the course of decoding aspatial segment of a dependent picture, all necessitated respective baselayer resources will be available when the co-located spatial segmentplus the signaled offset is completely decoded.

An embodiment for a syntax is shown in FIG. 42.

min_spatial_segment_delay describes the decoding delay of the currentlayer introduced by coding dependencies with respect to thecorresponding base layer in terms of spatial segments.

Depending on the value of min_spatial_segment_delay, the followingapplies:

If min_spatial_segment_delay is equal to 0, no limit on the minimumdelay between decoding of layers is signaled

Otherwise (min_spatial_segment_delay is not equal to 0), it is arequirement of bit stream conformance that exactly one of the followingconditions shall be true:

-   -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 0 and        entropy_coding sync_enabled_flag is equal to 0 (i.e. neither        tiles nor WPP are used in the video sequence), and all base        layer resources for the decoding process of any slice segment A        of the current layer in bit stream order are available when the        first slice segment C that follows (min_spatial_segment_delay        −1) slices in bitstream order in the base layer after the last        slice segment B in bitstream order that contains at least a part        of the same image area with respect to slice A in the current        layer is decoded completely.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 1 and        entropy_coding sync_enabled_flag is equal to 0 (i.e. tiles are        used in the video sequence), and all base layer resources for        the decoding process of any tile A of the current layer in        bitstream order are available when the first tile C that follows        (min_spatial_segment_delay −1) tiles in bitstream order after        the last tile B in bitstream order that contains at least a part        of the same image area with respect to tile A are completely        decoded.    -   In each picture parameter set that is activated within the coded        video sequence, tiles_enabled_flag is equal to 0 and        entropy_coding sync_enabled_flag is equal to 1 (i.e. WPP is used        in the coded video sequence), and all base layer resources for        the decoding process of the any CTB row A in the current layer        are available when the first CTB row C that follows        (min_spatial_segment_delay −1) CTB rows after the last CTB row B        of the base layer that covers at least part of the same image        area with respect to CTB row A of the enhancement layer in        bitstream order are completely decoded.

A picture, or sub-coded-video-sequence based signaling ofmin_spatial_segment_delay is feasible as well. The scope of the SEImessage in terms of associated NAL units is smaller than the coded videosequence in the temporal domain as given in FIG. 20 and is eitherdefined by the position of the SEI message in the bitstream or by anindex. One embodiment is given in the Layer_decoding_delay_SEI of FIG.43.

Semantics with respect to the former described embodiments may change toreflect the scope of the SEI message and its syntax elements.

The above explicit examples may be slightly revised. In the aboveexamples, the syntax element structure comprisesmin_spatial_segment_delay and ctb_based_delay_enabled_flag withmin_spatial_segment_delay either measuring, in a one-dimensional orscalar manner—along the spatial segment/CTB decoding order -, theinter-layer coding offset in units of spatial segments or CTBs dependingon ctb_based_delay_enabled_flag. But as the number of CTBs of the baselayer picture is generally greater than the number of spatial segmentssuch as tiles or substreams of the base layer picture, in a slightlydifferent embodiment, in case of ctb_based_delay_enabled_flag indicatingthe CTB-based inter-layer offset indication, this inter-layer offset isnot determined based on min_spatial_segment_delay only, but rather thelatter syntax element is, in that case, interpreted as indicating thelocation of that CTb of the base layer picture in the horizontaldimension, the complete decoding of which may be used as a trigger bythe decoder so as to commence decoding the enhancement layer picture.Naturally, min_spatial_segment_delay could alternatively be interpretedas indicating the base layer pictures's CTB along the verticaldimension. Depending on ctb_based_delay_enabled_flag, namely if sameindicates the CTB based indication, a further syntax element istransmitted in the data stream which indicates the location of the baselayer picture's CTB acting as the just mentioned trigger, in the otherdimension.

That is, the following syntax fragment may be used for signaling, i.e.may be used as the syntax element structure:

The indices i and j may indicate the layer id of the base layer and theenhancement layer.

min_spatial_segment_offset_plusl[ i ][ j ] ue(v) if(min_spatial_segment_offset_plus1[ i ][ j ] > 0 ) { ctu_based_offset_enabled_flag[ i ][ j ] u(1) if(ctu_based_offset_enabled_flag[ i ][ j ] ) min_horizontal_ctu_offset_plusl[ i ][ j ] ue(v)

The semantics of the above syntax elements may be described as follows:

min_spatial_segment_offset_plus1[i][j] indicates the spatial region, ineach picture of the j-th direct reference layer of the i-th layer, thatis not used for inter-layer prediction for decoding of any picture ofthe i-th layer, by itself or together withmin_horizontal_ctu_offset_plus1[i][j], as specified below. The value ofmin_spatial_segment_offset_plus1[i][j] shall be in the range of 0 torefPicWidthInCtbsY[i][j] *refPicHeightInCtbsY[i][j], inclusive. When notpresent, the value of min_spatial_segment_offset_plus1_[i][j] isinferred to be equal to 0.

ctu_based_offset_enabled_flag[i][j] equal to 1 specifies that thespatial region, in units of CTUs, in each picture of the j-th directreference layer of the i-th layer, that is not used for inter-layerprediction for decoding of any picture of the i-th layer is indicated bymin_spatial_segment_offset_plus1 [i][j] and min_horizontalctu_offset_plus1 [i][j] together. ctu_based_offset_enabled_flag[i][j]equal to 0 specifies that the spatial region, in units of slicesegments, tiles, or CTU rows, in each picture of the j-th directreference layer of the i-th layer, that is not used for inter-layerprediction for decoding of any picture of the i-th layer is indicated bymin_spatial_segment_offset_plus1_[i] only. When not present, the valueof ctu_based_offset_enabled_flag[i] is inferred to be equal to 0.

min_horizontal_ctu_offsetplus1[i][j], when ctu_based_offset_enabled_flag[i][j] is equal to 1, indicates the spatial region, in each picture ofthe j-th direct reference layer of the i-th layer, that is not used forinter-layer prediction for decoding of any picture of the i-th layer,together with min_spatial_segment_offset_plus1_[i][j], as specifiedbelow. The value of min_horizontal_ctu_offset_plus1[i][j] shall be inthe range of O to refPicWidthInCtbsY[i][j], inclusive.

When ctu_based_offset_enabled_flag[i][j] is equal to 1, the variableminHorizontalCtbOffset[i][j] is derived as follows:

-   minHorizontalCtbOffset[i][j]=(min_horizontal_ctu_offset_plus1[i][j]>0)?    (min_horizontal_ctu_offset_plus1 [i][j]−1):    (refPicWidthInCtbsY[i][j]−1)

The variables curPicWidthInSamplesa [i], curPicHeightInSamplesa [i],curCtblog2SizeY[i], curPicWidthInCtbsY[i], and curPicHeightInCtbsY[i]are set equal to PicWidthInSamples_(L), PicHeightInSamples_(L),Ctblog2SizeY, PicWidthInCtbsY, and PicHeightInCtbsY, respectively, ofthe i-th layer.

The variables refPicWidthInSamples_(L)[i][j], refPicHeightInSamplesL[i][j], refCtblog2SizeY[i][j],refPicWidthInCtbsY[i][j], and refPicHeightInCtbsY[i][j] are set equal toPicWidthInSamples_(L), PicHeightInSamplesL, Ctblog2SizeY,PicWidthInCtbsY, and PicHeightInCtbsY, respectively, of the j-th directreference layer of the i-th layer.

The variables curScaledRefLayerLeftOffset[i][j],curScaledRefLayerTopOffset [i][j], curScaledRefLayerRightOffset [i][j]and curScaledRefLayerBottomOffset[i][j] are set equal toscaled_ref_layer_left_offset[j]<<1, scaled_ref_layer_top_offset[j]<<1,scaled_ref_layer_right_offset[j]<<1, scaled_ref_layer_bottom_offset[j]<<1, respectively, of the j-th direct reference layer of the i-thlayer.

The variable colCtbAddr[i][j] that denotes the raster scan address ofthe collocated CTU, in a picture in the j-th direct reference layer ofthe i-th layer, of the CTU with raster scan address equal to ctbAddr ina picture of the i-th layer is derived as follows:

The variables (xP, yP) specifying the location of the top-left lumasample of the CTU with raster scan address equal to ctbAddr relative totop-left luma sample in a picture of the i-th layer are derived asfollows:

-   xP=(ctbAddr % curPicWidthInCtbsY[i]) <<curCtblog2SizeY-   yP=(ctbAddr/curPicWidthInCtbsY[i]) <<curCtblog2SizeY

The variables scaleFactorX[i][j] and scaleFactorY[i][j] are derived asfollows:curScaledRefLayerPicWidthInSamples_(L)[i][j]=curPicWidthInSamples_(L)[i]curScaledRefLayerLeftOffset[i][j]=curScaledRefLayerRightOffset[i][j]

curScaledRefLayerPicWidthInSamples_(L)[i][j]=curPicHeightInSamples_(L)[i]curScaledReflayerTopOffset[i][j]−curScaledRefLayerBottomOffset[i][j]scaleFactorX[i][j]=((ref PicWidthInSamplesL [i][j] <<16)+

-   (curScaledRefLayerPicWidthInSamples_(L) W[j]    >>1)/curScaledRefLayerPicWidthInSamples_(L)) [i][j]-   scaleFactorY[i][j]=((refPicHeightInSamplesL [i][j] <<16)+-   (curScaledRefLayerPicWidthInSamples >>1))/curScaledRefLayerPicWidthInSamples    [i][_(J])

The variables (xCol[I][j], yCol xCol[I][j]) specifying the collocatedluma sample location in a picture in the j-th direct reference layer ofthe luma sample location (xP, yP) in the i-th layer are derived asfollows:

-   xCol[i][j]=Clip3(0, (refPicWidthInSamplesL[i][j]−1),-   ((xP−curScaledRefLayerLeftOffset[i][j]) * scaleFactorX[i][j] +(1    <<15)) >>16)) yCol [i][j]=Clip3(0, (refPicHeightInSamplesL    [i][j]−1),-   ((yP−curScaledRefLayerTopOffset[i][j]) * scaleFactorY[i][j]+(1<<15))    >>16))

The variable colCtbAddr[i][j] is derived as follows:

-   xColCtb[i][j]=xCol[i][j]>>refCtblog2SizeY[i][j]-   yColCtb[i][j]=yCol[i][j]>>refCtblog2SizeY[i][j]-   colCtbAddr[i][j]=xColCtb[i][j]+(yColCtb[i][j] *    refPicWidthInCtbsY[i][j])

When min_spatial_segment_offset_plus1 [i][j] is greater than 0, it is arequirement of bitstream conformance that the following shall apply:

If ctu based offset enabled flag[i][j] is equal to 0, exactly one of thefollowing applies:

-   -   In each PPS referred to by a picture in the j-th direct        reference layer of the i-th layer, tiles_enabled_flag is equal        to 0 and entropy_coding sync_enabled_flag is equal to 0, and the        following applies:    -   Let slice segment A be any slice segment of a picture of the        i-th layer and ctbAddr be the raster scan address of the last        CTU in slice segment A. Let slice segment B be the slice segment        that belongs to the same access unit as slice segment A, belongs        to the j-th direct reference layer of the i-th layer, and        contains the CTU with raster scan address colCtbAddr[i][j]. Let        slice segment C be the slice segment that is in the same picture        as slice segment B and follows slice segment B in decoding        order, and between slice segment B and that slice segment there        are min_spatial_segment_offset_plus1_[i]−1 slice segments in        decoding order. When slice segment C is present, the syntax        elements of slice segment A are constrained such that no sample        or syntax elements values in slice segment C or any slice        segment of the same picture following C in decoding order are        used for inter-layer prediction in the decoding process of any        samples within slice segment A.    -   In each PPS referred to by a picture in the j-th direct        reference layer of the i-th layer, tiles_enabled_flag is equal        to 1 and entropy_coding sync_enabled_flag is equal to 0, and the        following applies:    -   Let tile A be any tile in any picture picA of the i-th layer and        ctbAddr be the raster scan address of the last CTU in tile A.        Let tile B be the tile that is in the picture picB belonging to        the same access unit as picA and belonging to the j-th direct        reference layer of the i-th layer and that contains the CTU with        raster scan address colCtbAddr[i][j]. Let tile C be the tile        that is also in picB and follows tile B in decoding order, and        between tile B and that tile there are        min_spatial_segment_offset_plus_1_[i] −1 tiles in decoding        order. When slice segment C is present, the syntax elements of        tile A are constrained such that no sample or syntax elements        values in tile C or any tile of the same picture following C in        decoding order are used for inter-layer prediction in the        decoding process of any samples within tile A.    -   In each PPS referred to by a picture in the j-th direct        reference layer of the i-th layer, tiles_enabled_flag is equal        to O and entropy_coding sync_enabled_flag is equal to 1, and the        following applies:    -   Let CTU row A be any CTU row in any picture picA of the i-th        layer and ctbAddr be the raster scan address of the last CTU in        CTU row A. Let CTU row B be the CTU row that is in the picture        picB belonging to the same access unit as picA and belonging to        the j-th direct reference layer of the i-th layer and that        contains the CTU with raster scan address colCtbAddr[i][j]. Let        CTU row C be the CTU row that is also in picB and follows CTU        row B in decoding order, and between CTU row B and that CTU row        there are min_spatial_segment_offset_plus1 [i] −1 CTU rows in        decoding order. When CTU row C is present, the syntax elements        of CTU row A are constrained such that no sample or syntax        elements values in CTU row C or row of the same picture        following C are used for inter-layer prediction in the decoding        process of any samples within CTU row A.

Otherwise (ctu based offset enabled flag[i][j] is equal to 1), thefollowing applies:

-   -   The variable refCtbAddr[i][j] is derived as follows:        xOffset[i][j]=((xColCtb[i][j]+minHorizontalCtbOffset[i][j])>(refPicWidthInCtbsY[i][j]))?        (refPicWidthInCtbsY[i][j]−1-xColCtb[i][j]):        (minHorizontalCtbOffset[i][j])        yOffset[i][j]=(min_spatial_segment_offset_plus1[i][j] −1) *        refPicWidthInCtbsY[i][j]        refCtbAddr[i][j]=colCtbAddr[i][j]+xOffset[i][j] +yOffset[i][j]    -   Let CTU A be any CTU in any picture picA of the i-th layer, and        ctbAddr be the raster scan address ctbAddr of CTU A. Let CTU B        be a CTU that is in the picture belonging to the same access        unit as picA and belonging to the j-th direct reference layer of        the i-th layer and that has raster scan address greater than        refCtbAddr[i][j]. When CTU B is present, the syntax elements of        CTU A are constrained such that no sample or syntax elements        values in CTU Bare used for inter-layer prediction in the        decoding process of any samples within CTU A.

That is, summarizing the just-presented embodiment, the CTB-basedindication of the inter-layer offset may be done in a switchable wayusing a flag which switches between this CTB-based indication on the onehand the spatial segment based indication of the inter-layer offset onthe other hand and in doing so, the CTB-based alternative may use thesyntax element which transmitted unconditionally, i.e. irrespective ofthe CTB-based or spatial segment based inter-layer offset indication,that is the syntax element indicating the inter-layer offset in units ofspatial segments in case of the spatial segment-wise indication beingsignaled, as one component, such as horizontal or vertical component, ofthe location of a “trigger CTB” in the base layer picture. Depending onthe syntax element switching between the CTB based and spatial segmentbased inter-layer offset indication, a further syntax element is thentransmitted or not. In particular, it is transmitted if the CTB basedindication is signaled. In that case, the latter syntax elementindicates the missing dimension of the location of the “trigger CTB”.The decoder thus may identify the “trigger CTB” among the regulararrangement of CTBs of the base layer picture in rows and columns usingthe two syntax elements and as soon as this CTB has been decoded, thedecoder may commence decoding the enhancement layer picture. Still, theindication of any inter-layer offset may be completely switched off byway of using one of the representable states of the first syntaxelement, i.e. min_spatial_segment_delay. Owing to the predetermineddecoding order defined among CTBs, in the CTB-based inter-layer offsetindication case, the decoder may still transfer the horizontal andvertical components of the position of the trigger CTB into a number ofCTBs of the base layer picture which have to be completely decodedbefore commencing the decoding of the first CTB of the enhancement layerpicture so that the decoder may use the embodiment of FIG. 36 so as tocontrol the abeyance of the CTB-based indication of the inter-layeroffset.

In the following, a further aspect of the present invention is describedin more detail. The fourth aspect is about the problem that allparticipating network entities which receive the bitstream from theencoder are enabled to easily distinguish between the various layersconveyed in the multi-layer data stream. For example, intermediatenetwork entities may be interested in excluding certain informationlayers from further transmission such as layers concerning sampleresolutions exceeding some resolution threshold. The followingdescription provides an overview of the current situation in theenvisaged extension of HEVC.

The Video Parameter Set (VPS) of HEVC [1] provides high level access toa coded bitstream and contains information that is vital to processbitstreams at an intermediate or end device. The upcoming scalable andmultiview extensions of HEVC will further benefit from the VPS extensionthat provides syntax designed for scalable bitstreams. One of the maintasks of the VPS extension is to provide a unified approach tointerpretation of the nuh reserved_zero_6 bits in the NAL unit header.The nuh_reserved_zero_6 bits are intended to be relabeled as layer_idand serve as a generic layer identifier in a scalable video codingscenario. The layer_id syntax element in the NAL unit header is given infollowing table along the NAL unit header given in [1] and as shown inFIG. 22.

Two general approaches were considered in the design process [5].Firstly, an approach that maps values of a single identifier in theheader of NAL units to potentially multiple scalable identifiers assignaled in the VPS extension. Secondly, an approach that assignsindividual bits (or blocks of bits) of a single identifier in the headerof NAL to specific scalable identifiers signaled in the VPS extension.

The design of the current VPS extension syntax as reported in [4] usesthe mapping approach but already contains all syntax elementsnecessitated for both approaches, namely two syntax elements indicatethe types of scalability (cp. scalability_map) and the amount of layersper scalability dimension (cp. dimension_id_len_minus1).

The mapping approach introduces further syntax elements to the VPSextension syntax, i.e. the actual value of the scalable identifier codedas u(v) and optionally the layer_id if an encoder chooses to allocatethe values of layer_id sparsely in a non-consecutive form.

In many very likely scalable scenarios, e.g. two or three spatial layer,two or three views and others, it is not necessitated to utilize all 63values of the 6 bit layer identifier in the NAL unit header. For thesescalable scenarios, the approach that assigns individual bits of thelayer identifier in the NAL unit header to specific scalable dimensionhas two advantages when compared to a mapping-based approach:

-   -   Interpretation of layer identifier values in the NAL unit header        in terms of scalability dimensions identifiers necessitates no        indirection or look up.    -   VPS extension syntax elements necessitated for mapping approach        do not need to be transmitted which account for a significant        portion of the VSP extension bits for scalability signaling.    -   Intermediate devices do not need to store mapping table for each        passing video bitstream.

In accordance with the concept of the fourth aspect described below,there may be a hint within the High Level Syntax of HEVC that indicateswhether a mapping or a partitioning approach is used.

According to one embodiment, depending on the hint, a portion of themapping related syntax elements (cp. vps_nuh_layer_id_present flag,layer_id_in_nuh[i] and dimension_id[i][j]) is either transmitted or leftout and the syntax elements for scalability types (cp. scalability mask)and the amount of layers per scalability (cp. dimension_id_len_minus1),are signaled and have to be interpreted according to the hint either asinformation about the partitioning or the mapping of the scalableidentifier in the NAL unit header.

An embodiment corresponding to, or exploiting, the concept of the fourthaspect of the present invention is presented with respect to FIG. 23.FIG. 23 shows a network entity which may be a video decoder as any oneof those already discussed above, or may be an intermediate networkentity resulting between encoder and decoder. The network entity isgenerally indicated using reference sign 680. It is for processing amulti-layer video data stream 682, such as for example any one of thedata streams 40 mentioned above. In the case of network entity 680 beinga video decoder, the processing would involve decoding the multi-layervideo data stream 682. In the case of an intermediate network entity,the processing could, for example, encompass a forwarding of the videodata stream.

A scene is coded into the multi-layer video data stream in layers sothat, in each layer, this scene is coded in a different operationalpoint of a scalability space spanned by scalability dimensions, whereinthe multi-layer video data stream is composed of first NAL units, eachof which is associated with one of the layers, and second NAL unitsinterspersed within the first NAL units and presenting generalinformation on the multi-layer video data stream. In other words, thefirst NAL units 684 may carry one or more slices of pictures of thevideo, with the “picture” corresponding to any of the layers of themulti-layer video data stream 682. In the above embodiments, for ease ofdescription, merely two layers were discussed, layer 0 and layer 1.Naturally, the number of layers may be greater than two and even thesort of information which a layer contributes to any of the previouslayers may differ from layer to layer. Besides the first NAL units 684,the NAL units 686 are shown to be interspersed between NAL units 684,but their transmission may be performed by way of a separate channelcompared to the first NAL unit 684. The second NAL units present generalinformation on the multi-layer video data stream in the manner set outin more detail below.

In order to describe the association between the first NAL units on theone hand and the set of layers of data stream 682, on the other hand, inmore detail, reference is made to FIG. 24. FIG. 24 shows a first NALunit 684 as a representative for all first NAL units 684, It comprises,within a header 688, a layer indicator field 690. Besides header 688,NAL unit 684 comprises payload data 692 concerning slice data asdiscussed above with respect to the other figures, i.e. data concerningthe video content encoded using inter-layer prediction. FIG. 24 alsoshows the set of layers, namely 694. In particular, the set 694 oflayers shown in FIG. 24, shall denote all of the possible layers whichmay be represented and distinguished from each other by way of the layerindicator field 690 in NAL unit 684. That is, the association functionbetween set 694 on the one hand, and the possible values of the layerindicator field 690, shall be assumed as being a bijective one. In FIG.24, the individual layers of set 694 are exemplarily illustrated usingsmall circles, each having a certain number inscribed thereinto.Although these inscribed numbers suggest an order to be defined amongthe layers of set 694, it should be noted that the following discussionwill reveal that the way the layers in set 694 are arranged or sorted isnot derivable on the basis of the layer indicator field 690, only.Rather, to this end, the network entity 680 needs to inspect a typeindicator field 696 in the interspersed second NAL units 686. However,this is described later.

In other words, until now each element of set 694 in FIG. 24 merelyrepresents one of the possible states of the layer indicator field 690in the NAL units 684. The layers in set 694 are distinguishable by wayof the layer indicator field 690, but the semantic meaning and orderamong these layers does not become clear for the network entity 680without additional information provided by the second NAL unit 686.Nevertheless, in reality the layers of set 694 form nodes of a tree withthe branches between the trees corresponding to certain scalabilitydimensions or axes. One of the layers is, for example, the base layerand corresponds to the root of the tree. Each branch connecting twolayers, i.e. two nodes of the tree, describes how a certain layercontributes to another layer, i.e. which sort of information isadditionally provided using inter-layer prediction, the sort ofinformation corresponding to the scalability dimension and may be, forexample, spatial resolution increase, SNR increase, or the like. For thesake of simplicity, FIG. 24 shows a two-dimensional scalability space698 spanned by two scalability dimensions 700 and 702 corresponding to,for example, spatial scalability and SNR scalability. An exemplary treeof layers and its extension through space 698 is shown in FIG. 24. FIG.24 also shows that not all of the possible layers of set 694 may bepresent in data stream 682. In the exemplary case of FIG. 24, forexample, merely five layers are actually used in the data stream 682.

FIG. 22, for example, showed that the layer indicator field may have 6bits, thereby distinguishing 2⁶ =64 possible states or possible layersof set 694. It is the mapping from these possible values or layers ofset 694 onto the operational points in the scalability space 698 whichmay be adjusted in a manner described in more detail below by way of thesecond NAL unit 686. The mapping is indicated in FIG. 24 using referencesign 704. The “operational points” shall denote the positions of atleast the actually present layers within set 694 within the scalabilityspace 698. For example, the origin of scalability space 698 may beassociated with the base layer or tree root, while each branch along anyof axes 700 and 702 of scalability space 698 may have a fixed length of1. The vectors pointing to the operational points in scalability space698 may thus have integer valued coordinates.

Briefly summarizing the description brought forward so far, themulti-layer video data stream 682 provides information about a videocontent or scene in multiple layers. The layers are arranged in a treewith each layer being connected to the tree via a branch. Starting froma base layer forming the tree root, a next following layer contributesto the reconstructible version of the video content informationconcerning a certain kind of information, interpretable as scalabilitydimension. Accordingly, each layer is either the tree root layer or isconnected to the latter through a certain path of branches and the NALunits 684 belonging to layers lying along this path are needed so as toreconstruct the video content at the respective layer. Naturally, it isadvantageous if the mapping 704 is performed such that any“contributing” layer leading at a distal end of a respective branch withrespect to the tree root has a value of the layer indicator field 690,which is higher than the value of the layer indicator field of the layerat the proximal end of the respective branch.

FIG. 25 shows the meaning of the type indicator field 696 in moredetail. FIG. 25 shows the layer indicator field 690 as a field of fixedbit length. In any case, the length of field 690 is independent from thevalue of the type indicator field 696. However, if the type indicatorfield has a first state, the layer indicator field 690 is treated as awhole, i.e. all of its n bits are treated commonly to distinguishpossible values thereof. Advantageously, a n-bit integer number isderived from the layer indicator field 690 by network entity 680 if thetype indicator field 696 assumes the first state. In the case of typeindicator field 696 assuming the first state, network entity 680performs the mapping 704 of the possible values of the m-bit field 690onto the operational points by use of mapping information conveyedwithin the second NAL units 686, the mapping information being indicatedin FIG. 25 using reference sign 708. As shown in FIG. 25, the mappinginformation 708 comprises, for example, a table assigning to eachactually used possible value of the layer indicator field 690 a vectorpointing to the associated operational point of the respective possiblevalue. That is, in case of the type indicator field 696 assuming thefirst state, the network entity 680 may derive the mapping information708 from the second NAL units 686 and may perform for each layerindicator field 690 a look-up into the mapping information or table 708so as to look-up the associated vector so as to locate therespective/associated operational point in space 698. The number p ofdimensions of the vectors associated to the possible values in mappinginformation 708 may be set by default or may be signaled within the datastream, such as the second NAL units 686. Later on, it will be shownthat the following information may be signaled in order to conveyinformation about the mapping information 708: vps-_max_layers_minus1may determine the number M of actually used possible values of the M-bitfield 690. num_dimensions_minus1 may define the numbers of dimensions.The latter two variables may be transmitted using the syntax elementscalability mask. The table itself may then be signaled via pairs ofpossible values on the one hand, namely layer_id_in_nuh, and ap-dimensional vector, namely dimension_id[i][j]. The mapping 704 thenleads to a p-dimensional vector, namely the one mapped via the mappinginformation 708 onto this respective vector 710 pointing to theoperational point within space 698 associated with the layer of the NALunit having layer indicator field 690.

If, however, the type indicator field 696 assumes a second state, themapping 704 is performed differently. In particular, the mapping isperformed in that case by splitting the layer indicator field 690 intomore than one portion, i.e. by splitting up the sequence of m-bits offield 690 into a number n of consecutive sub-sequences of bits. Eachportion thus obtained is used as a coordinate x₁. . . x_(n) of ann-dimensional vector 712, which in turn points to the operational pointwithin the scalability space 698, associated with the layer of the NALunit comprising the layer indicator field 690. For example, the m-bitsof the layer indicator field 690 are split up into n portions by forminga first portion out of the first (most significant) m1 bits of field690, a second portion out of the following m2 (next less significant)bits of field 690 and so forth up to the n-th portion with m₁+ . . .+m_(n)=m. Th bits of each portion may directly be treated as an integervalue representation.

In the case of the type indicator field assuming the second state, thenumber of dimensions n may be set by default or signaled via the datastream. In case of the specific embodiment described herein after, n isderived based on the same syntax element used in order to determine p incase of the type indicator field 696 assuming the first state, namelybased on scalability_mask. The bit length of the portions into which thelayer indicator field 690 is subdivided, namely m₁, . . . , m_(n), arealso exemplarily signaled via syntax elements dimension_id_len_minus1,but again, the subdivision may be achieved by default without explicittransmission.

It should be noted that the same syntax structure, such asscalability_mask, being parsable irrespective of the type indicatorfield 696, may indicate the number and semantic meaning of dimensions ofthe scalability space. It might be, that the allowed states of thissyntax element are restricted—relative to the mapping case—onto a propersubset of available possible states of that syntax element in case ofthe type indicator field 696 indicating the component-wiseinterpretation of the layer indicator field, however, since, forexample, the maximum number of dimensions of the scalability space beingavailable (or settable) in the mapping case (first state of typeindicator filed) might be higher than the maximum number available inthe component interpretation case (first state of type indicator filed).The encoder would obey this restriction accordingly.

Exemplary use cases for the embodiment of FIGS. 23 to 25 are:

Mass Multi Party Conferencing

In a mass conversational service scenario based on RTP, e.g. a webconference, transferred video between multiple parties is adapted in amultipoint control unit (MCU) that has to be aware of the parameter setsof the respective video bitstreams. Each party provides a thumbnailbitstream and two bitstreams with enhanced spatial resolution, e.g. 720pand 4K, of a speaker. The MCU takes decisions about which streams toprovide to which party. Easy parsing of scalability parameters istherefore a significant relief for the MCU. A partitioning basedapproach would necessitate less computational and memory resourcescompared to a mapping based approach of scalability signaling.

Transport Systems

In transport systems, such as RTP or MPEG2-TS, mapping the scalabilityrelated codec information to respective elements could benefit from aless complex and bit saving mechanism such as partitioning compared tothe mapping-based approach. It is unlikely that transport systems willadopt a scalability signaling in the fashion of a mapping approach butwill rather resolve the mapping indirection and produce dedicatedscalability identifiers for each scalability dimension which will inturn be explicitly signaled such as in the partitioning approach.

The example in FIG. 26 presents a possible embodiment as a syntax flagin the VPS extension syntax that allows switching between a mapping anda partitioning based approach for scalability signaling in theextensions of HEVC, based on the VPS extension syntax as reported in[4].

dedicated_scalability_ids_flag with a value equal to 1 indicates thatthe bits of the layer_id field in the NAL unit header are splitaccording to the values of dimension_id_len_minus_1_[] and belong to therespective scalability dimensions signaled in scalability_mask.dedicated_scalability_ids_flag with a value equal to 1 indicates thatthe syntax elements vps_nuh_layer_id_present_flag, layer id in nuh[i]and dimension_id[i][j] are not signaled. dedicated_scalability_ids_flagwith a value equal to 1 indicates that the derivation of variablesdescribing the scalable identifiers of the corresponding NAL unit useonly the respective associated bits of the scalable identifier in theNAL unit header (cp. layer_id), e.g. in a fashion as follows:

DependencyId=layer_id && Ox07

QualityId=layer id && Ox38

-   dedicated_scalability_ids_flag with a value equal to O indicates    that the syntax elements vps_nuh_layer_id_present_flag,    layer_id_in_nuh[i] and dimension_id[i][j] are signaled and the bits    of layer_id in the NAL unit header are not associated to a specific    scalability dimension but mapped to scalable identifiers in the VPS    extension. dedicated_scalability_ids_flag with a value equal to O    indicates that the derivation of variables describing the scalable    identifiers of the corresponding NAL unit uses the syntax elements    dimension_id[i][j] given layer_id is signaled in the NAL unit    header, e.g. in a fashion as follows:

if (layer_id==layer_id_in_nuh[0]){

DependencyId=dimension_id[0][0 ]

QualityId=dimension_id[0][1]

dimension_id_len_minus1[i] when dedicated scalability_ids_flag is equalto 0 indicates the length in bits of dimension_id[i][j].dimension_id_len_minus1_[i] when dedicated_scalability_ids_flag is equalto 1 indicates

the number of bits of layer_id in the NAL unit header associated withthe i-th scalability dimension as indicated by scalability_mask.

The flow chart in FIG. 27 further illustrates the possible embodiment ofthe invention. Scalable identifiers are either derived from the bits oflayer_id directly via a masked bit copy or signaled in the VPS viaassociation to a specific value of layer_id.

Another syntax example is shown in FIG. 31. Here, the type indicatorfiled is signaled by “splitting_flag”, while the layer indicator filedis called nuh_layer_id. Depending on “splitting_flag”, the mappingconcept or the partitioning concept is used for deriving the respectiveNAL unit's layer's operational point in scalability space from thenuh_layer_id. The partitioning concept is, exemplarily, signaled bysplitting_flag being equal to 1. Then, scalable identifiers, i.e. thevector components concerning the scalability dimensions of thescalability space, can be derived from the nuh_layer_id syntax elementin the NAL unit header by a bit masked copy. The respective bit mask forthe i-th scalable dimension, the i-th component of vector 712 in FIG.25, is defined as follows. In particular, splitting_flag equal to 1indicates that the dimension_id[i][j] syntax elements are not present(i.e. no mapping information 708 is present) and that the binaryrepresentation of the nuh_layer_id value in the NAL unit header is splitinto a sequence of NumScalabilityTypes, i.e. n, segments, x_(1 . . . n)with lengths, in bits, according to the values of dimensio_id_len_minus1[j] and that the values ofdimension_id[LayerIdxInVps[nuh_layer_id]][j], i.e. the componentsx_(1 . . . n) of vector 712, are inferred from the NumScalabilityTypessegments of field 690. The semantic meaning and number of scalabilityaxes of the scalability space is signaled by a sequence of flagsscalability_mask_flag indicating for each of a here exemplarily fixednumber of predefined scalability types as to whether the respectivescalability type belongs to any of the scalability dimensions of thescalability space 698 or not. In particular, network entity 680 is ableto derive the scalability space, i.e. semantic meaning and number ofscalability axes NumScalabilityTypes, from the sequence of flagsscalability_mask_flag according to the for loop in FIG. 31:

  for( i = 0, NumScalabilityTypes = 0; i < 16; i++ ) { scalability_mask_flag[ i]  NumScalabilityTypes +=scalability_mask_flag[ i]  }where scalability_mask_flag[i] equal to 1 indicates that the i-thscalability dimension is present. and scalability_mask_flag[i] equal to0 indicates that the i-th scalability dimension is not present. Here,i=1 may denote MultiviewView scalability, i=2 may denote spatial/SNRscalability and i=0 may denote the addition of depth map information.Other scalability dimension types may exist as well and, naturally, thejust outlined example is merely illustrative. Assuming the length ofnuh_layer_id to be exemplarily be 6, the partitioning or masked copy maybe done as follows:

The variable dimBitOffset[0] is set equal to 0 and for j in the range of1 to NumScalabilityTypes −1, inclusive, dimBitOffset[j] is derived asfollows:

${{dimBitOffset}\;\lbrack j\rbrack} = {\sum\limits_{{dimIdx} = 0}^{j - 1}\left( {{{dimension\_ id}{\_ len}{{\_ minus1}\lbrack{dimIdx}\rbrack}} + 1} \right)}$

The value of dimension id_len_minus1[NumScalabilityTypes −1] is inferredto be equal to 5—dimBitOffset[NumScalabilityTypes −1].

The value of dimBitOffset[NumScalabilityTypes] is set equal to 6.

It is a requirement of bitstream conformance that whenNumScalabilityTypes is greater than 0, dimBitOffset[NumScalabilityTypes−1] shall be less than 6.

For j from O to NumScalabilityTypes −1, inclusive, dimension_id[i][j] isinferred to be equal to ((nuh_layer_id & ((1<<dimBitOffset[j+1])−1))>>dimBitOffset[j]).

Summarizing, the following syntax elements are involved with associatingeach first NAL unit with its operational point in the scalability space:

-   -   1) nuh_layer_id i.e. layer indicator filed 690    -   2) the sequence of scalability_mask_flag, i.e. information        revealing number and meaning of scalability axes 700, 702 of        space 698, and thus the number n of portions x_(i) of field    -   3) dimension_id_len_minus1, i.e. the bit length, of each portion        xi of field 690 for each of the axes (all except one, because        the remaining one may be inferred to encompass all remaining        bits 706 of field 690    -   4) vps_max_layers_minus_1 is transmitted, in accordance with an        optional embodiment of FIG. 31, also, this syntax element        indicating the number of used/actual layers among the possible        Σ_(i) 2^(m) ^(i) ones, along with a sequence of        vps_max_layers_minus1 possible values of the partitioned layer        indicator field, i.e. a sequence of layer_id_in_nuh[i], thereby        defining a rank order among the possible operational points

In case of splitting_flag being equal to 0, the mapping concept is used.The mapping information 708 is then transmitted using the followinginformation:

-   -   1) the sequence of scalability_mask_flag, i.e. information        revealing number and meaning of scalability axes 700, 702 of        space 698, and thus the number p of components of the M vectors        in table 708    -   2) dimension_id_len_minus1, i.e. the bit length, of each        component x_(j) of the vectors dimension_id[i]u] in table 708,        i.e. one for each of the axes of space 698    -   3) layer_id_in_nuh[i] serve, optionally, as indices for the list        of M vectors dimension_id[i][j]    -   4) vps_max_layers_minus1 is optionally transmitted also although        not shown in FIG. 31, indicating the number M of used/actual        layers among the possible Σ₁ 2^(dimension id len minus1[i])ones

Thus, in case of splitting_flag being equal to 0, the vectors 710 arederived inherently by partitioning without explicit signaling, i.e.without signaling dimension_id[i][j] which is inferred instead.

Thus, in accordance with the fourth aspect, i.e. the concept of“switchable interpretation of NAL unit layer identifier” the bitstream40 may comprise NAL units, namely VPS NAL units, which comprise a typeindicator filed 300 via which it is possible to switch between themapping concept and the bit-splitting concept to interpret the layerindicator field 302 in the “normal” NAL units. Thus, the same bitpositions of the field 302 are used in both modes, but theinterpretation of the bits and the interpretation prescriptioninformation signaled, namely the mapping information or the split andsemantic information, transferred in the bitstream changes between themodes. Although this necessitates the additional transmission of thetype indicator field, this concept, altogether, leads to a moreefficient bitstream transmission as the advantages of the mappingconcept and the bit-splitting concept can be exploited as needed becauseboth concepts are differently suitable for different multi-layer datasuch as, depending on the number of layers or the like.

The fifth aspect of the present application concerns a multi-standardmulti-layer video decoder interface. The concept set out hereinafterdescribes an transport layer decoder and the interface between thetransport layer decoder (like MPEG transport stream or RTP) and ascalable video decoder that supports different video coding standards indifferent layers (e.g. H.264/AVC in the base layer and HEVC inenhancement layers).

A scalable video bitstream is composed of layers: It consists of onebase layer, which contains an independently decodable video signal andone or more enhancement layers which can only be decoded in combinationwith the base layer (and potentially other enhancement layers) andprovide a higher temporal resolution (temporal scalability), spatialresolution (spatial scalability), quality (SNR scalability), higher bitdepth (bit depth scalability) video signal or other camera views(multiview scalability).

Existing scalable video coding standards like H.264/AVC SVC define bothbase and enhancement layers in the same standard. They are designed in away, that a scalable bitstream has the same basic format as anon-scalable bitstream. If a scalable bitstream is input into anon-scalable decoder, it can still find the packet type and discard thepackets that are unknown.

HEVC is the first video coding standard allows using a different videocoding standard for the base layer (e.g. H.264/AVC). The packet formatsfor both standards are different, so a base layer decoder cannotunderstand the enhancement layer packets. On the other hand theenhancement layer decoder can understand the enhancement layer packetformat, but not the base layer packet format.

In audio/video systems a transport layer is used to combine severalaudio and video streams and to provide metadata like timing and streamtype.

In existing multi-layer transport layer decoders the access units ofbase and enhancement layer are multiplexed into a single video datastream (e.g. Annex B Byte stream of H.264/AVC). This video stream is theinput into the video decoder.

If different video coding standards are used for base and enhancementlayer, base and enhancement layer packets cannot be composed into asingle bitstream in all cases.

In accordance with an embodiment of the fifth aspect, a transport layerdecoder distinguishes the following cases:

-   -   1. the video decoder reading the output can only decode the base        layer    -   2. the video decoder reading the output can decode base and        enhancement layers and base and enhancement layers are encoded        using the same video coding standard    -   3. the video decoder reading the output can decode base and        enhancement layers and base and enhancement layers are encoded        using different video coding standards

In case 1 the transport layer decoder uses the following behavior:

Only the packets containing the base layer are extracted from thetransport layer and input into the single-standard single-layer videodecoder in a format specified in the video coding standard.

A specific embodiment is a transport stream decoder that extractsH.264/AVC NAL units from an MPEG-2 transport stream by selecting onlythe stream that has the stream type “AVC video stream conforming to oneor more profiles defined in Annex A of ITU-T Rec. H.264 |ISO/IEC14496-10 or AVC video sub-bit stream” assigned and inputs them to theH.264/AVC video decoder in the byte stream format defined in Annex B ofthe H.264/AVC specification. NAL units belonging to streams whose streamtype is not equal to the stream type “AVC video stream conforming to oneor more profiles defined in Annex A of ITU-T Rec. H.264|ISO/IEC 14496-10or AVC video sub-bit stream” are discarded within the transport streamdecoder. FIG. 28 gives an illustration of a specific embodiment.

In case 2 the transport layer decoder uses the following behavior:

Packets from base and enhancement layers are extracted from thetransport layer and input into the single-standard multi-layer videodecoder in a format specified in the video coding standard.

A specific embodiment could extract H.264/AVC NAL units from an MPEG-2transport stream by selecting the base layer stream that has the streamtype “AVC video stream conforming to one or more profiles defined inAnnex A of ITU-T Rec. H.264|ISO/IEC 14496-10 or AVC video sub-bitstream” assigned and also one or more enhancement layer streams thathave the stream type “SVC video sub-bit stream of an AVC video streamconforming to one or more profiles defined in Annex G of ITU-T Rec.H.264|ISO/IEC 14496-10” assigned. The NAL units of the different layerare multiplexed into the byte stream format defined in Annex B of theH.264/AVC specification and input into the H.264/AVC SVC video decoder.FIG. 29 gives an illustration of a specific embodiment.

In case 3 the transport layer decoder uses the following behavior:

Packets from base and enhancement layers are extracted from thetransport layer. These packets are handed to the multi-standardmulti-layer video decoder in one of the ways described in the followingsection.

Interface A

If the enhancement layer standard packet format allows carrying baselayer packets, the base layer packets are encapsulated in theenhancement layer format. This means a header is added to each baselayer packet that can be understood by the enhancement layer standardvideo decoder and which allows the enhancement layer video decoder toidentify the package as base layer of a different video coding standard.

In a specific embodiment of the invention the H.264/AVC NAL unit is usedas payload of an HEVC NAL unit, i.e. an HEVC NAL unit header is added infront of the H.264/AVC NAL unit header. A field in the HEVC NAL unitheader (e.g. nal_unit_type) is used to identify the payload as NAL unitof H.264/AVC. The bitstream can be input into the video decoder in HEVCAnnex B byte stream format. FIG. 30 gives an illustration of a specificembodiment.

Interface B

A different channel is used for each layer of the scalable bitstream.The video coding standard is determined in the decoder by the channelselection.

In a specific embodiment of the invention for two layers are handed overin two separate channels. The first channel is used only for H.264/AVCbase layer packets (or Annex B byte stream) while the second channel isused only for HEVC enhancement layer packets.

Interface C

A metadata field indicating the type of the video coding standard isassociated with each packet passed from the transport stream decoder tothe multi-standard multi-layer video decoder. Other metadata, like e.g.timing can be signaled in the same way.

In a specific embodiment, each base layer NAL unit is identified asH.264/AVC NAL unit by the associated metadata field and each enhancementlayer NAL unit is identified as HEVC NAL unit by the associated metadatafield.

Thus, the concept of the fifth aspect may be described as a“Multi-Standard Multi-Layer Video Decoder Interface” providingpossibilities as to how to combine bitstreams of different codecs.

Thus, in accordance with the fifth aspect of the present application, atransport layer decoder may be configured as outlined in the followingwith respect to FIG. 44. The transport layer decoder shown in FIG. 44 isgenerally indicated using reference sign 770. The transport layerdecoder 770 is configured to subject an inbound multi-layer video datastream 40 into which a scene is coded in layers, to decoding by amulti-standard multi-layer decoder 772 with which the transport layerdecoders 770 output interface is connected. The multi-layer video datastream 40 is composed NAL units as was already outlined above withrespect to the various other aspects of the present application, whichdescription shall be transferrable onto the embodiment of Fig. Each ofthe NAL units is associated with one of the layers. The layers areassociated with different codecs, i.e. with different standards. Foreach layer, the NAL units associated with the respective layer are codedusing the same codec, namely the one with which the respective layer isassociated.

The transport layer decoder 770 is configured to identify, for each NALunit, which codec same is associated with, and handover the NAL units ofthe multi-layer video data stream 40 to the multi-standard multi-layerdecoder which uses inter-layer prediction between layers associated withdifferent codecs to decode the multi-layer video data stream.

As mentioned above, each NAL unit may be associated with one of thelayers of the multi-layer video data stream 40 by way of a certain layerindicator field as already outlined above with respect to the fourthaspect of the present application. Some, or most, of the NAL units maycarry content related data, namely one or more slices. By collecting allNAL units concerning a certain set of layers, the video content or scenecoded into data stream 40 may be decoded by decoder 772 at aninformation amount conferred by this set of layers. With regard to thelayer dependency, the option of having more than one scalabilitydimension and so forth, reference is made to the description of thefourth aspect of the present application.

The multi-layer multi-standard decoder 772 is able to handle differentcodecs/standards. Examples for different standards have been presentedabove, i.e. H.264, HEVC, but other standards may mixed up as well. Thedifferent codecs/standards are not restricted to hybrid codecs. Rather,a mixture of different kinds of codecs may be used as well. Theinter-layer prediction used by the multi-layer multi-standard decoder772 may relate to prediction parameters used in the different layers, ormay refer to the picture samples of the various time-aligned layers.This has already been described above with respect to the other aspectsand embodiments.

The transport layer decoder 770 may be configured to perform the handingover of NAL units belonging to layers of codecs which the multi-layermulti-standard decoder 772 is able to cope with, only. That is, thehanding over performed by the transport layer decoder 770 may depend onthe transport layer decoder's 770 identification of the codec to whicheach NAL unit is associated. In particular, the transport layer decoder770 may perform the following for each NAL unit:

-   -   Identify the layer which the NAL unit currently inspected is        associated with by inspecting, for example, the layer indicator        field in the NAL unit's NAL unit header.    -   Based on an association between the layers of data stream 40 on        the one hand and the codecs/standards of which same are, which        association is derived by the transport layer decoder 770 based        on an inspection of respective high level syntax of data stream        40, the transport layer decoder 40 determines whether the        currently inspected NAL unit fulfills two criteria: the NAL unit        layer belongs to the subset of layers to be forwarded to decoder        772, which subset is determined by the currently inspected NAL        units layers operational point in scalability space and an        external prescription as to which operational points within the        scalability space are allowed to be forwarded to the multi-layer        multi-standard decoder 772 and which are not. Further, the        transport layer decoder 770 checks whether the currently        inspected NAL units layers codec belongs to the set of        codecs/standards which the multi-layer multi-standard decoder        772 is able to deal with.    -   If the check reveals that both criteria are fulfilled by the        currently inspected NAL unit, then the transport layer decoder        770 forwards the current NAL unit to decoder 772 for being        decoded.

Different possibilities exist for transport layer decoder 770, todetermine the above mentioned association between layers contained indata stream 40 on the one hand the codecs/standards underlying the sameon the other hand. For example, as discussed above with respect to“interface B”, different channels may be used in order to convey thedata stream 40, namely NAL units of layers of one codec/standard on onechannel, and NAL units of layers coded in accordance with anothercodec/standard on another channel. By this measure, the transport layerdecoder 770 is able to derive the just-mentioned association betweenlayers on the one hand and codecs/standards on the other hand bydistinguishing between the various channels. For example, the transportlayer decoder 770 determines, for each NAL unit of data stream 40, thechannel on which the respective NAL unit arrives so as to identify thecodec/standard with which the respective NAL unit or the respective NALunit's layer is associated.

Additionally or alternatively, the transport layer decoder 770 mayforward NAL units of layers belonging to different codec/standards tothe multi-layer multi-standard decoder 772 in a manner depending on therespective codecs/standards so that NAL units of layers belonging to onecodec/standard are transmitted on one channel to decoder 772, and NALunit of layers of a different codec/standard are forwarded tomulti-layer multi-standard decoder 772, on another channel.

The “different channels” may be provided by the underlying transportlayer. That is, the distinguishing between different channels may berealized by distinguishing between different channel identifiers asprovided by the underlying transport layer, which is not shown in FIG.44 for ease of understanding.

Another possibility of handing over the data stream 40 to multi-layermulti-standard decoder 772 would be that transport layer decoder 770encapsulates those NAL units having been identified as being associatedwith a layer which is associated with any codec different from apredetermined codec, using NAL unit headers of the predetermined codechaving a NAL unit type indicator set to a state indicating the codec ofthe respective layer. This means the following: the predetermined codecmay be, for example, any codec of any enhancement layer of data stream40. The base layer codec, for example, i.e. the codec associated withthe base layer of data stream 40, may be different from thepredetermined codec which is, for example, HEVC. Accordingly, transportlayer decoder 770 may turn data stream 40 into a data stream conformingto the predetermined codec when passing on the same to multi-layermulti-standard decoder 772. To this end, the transport layer decoder 770encapsulates each NAL unit belonging to a layer not coded using apredetermined codec, using NAL unit headers of the predetermine codecand sets aNAL unit type indicator within that_NAL unit header, to astate indicating a codec of the respective actual layer. For example,base layer NAL units are, for example, of H.264 and NAL units of thebase layer would, accordingly, be encapsulated using HEVC and NAL unitheaders having the NAL unit type indicator set to a state indicatingH.264. The multi-layer multi-standard 772 would accordingly receive anHEVC conforming data stream.

Naturally, as described with respect to interface C, it would bealternatively possible that transport layer decoder 770 provides eachNAL unit of the inbound data stream 40 with metadata indicating thecodec with which the layer associated with the respective NAL unit isassociated. The data stream's 40 NAL units would accordingly beforwarded to decoder 772 in a thus extended manner.

Using the alternatives just described, it is feasible to extend acontent encoded into a data stream by further layers which, however, arecoded using another codec such as, for example, a newer codec, and theexisting part of the codec 40 does not need to be amended. Multi-layermulti-standard decoders in turn, which are able to deal with the newercodec, i.e. the newly added one, are able to treat such mixtures ofexisting data streams with layers coded using the newer codec.

Thus, parallel/low delay video coding concepts for HEVC scalablebitstreams were presented above.

The High Efficiency Video Coding (HEVC) standard [1] initially featurestwo dedicated parallelization tools that allow parallel processing onencoder and decoder side: tiles and wavefront parallel processing (WPP).These tools allow parallelization within a picture that targetprocessing time improvements while minimizing coding efficiency losswhen compared to HEVC coded video that does not feature parallelprocessing within individual pictures.

In a scalable [2] or multiview [3] HEVC bitstream, a decoded base layeror base view picture is used to predict the enhancement layer ordependent view picture. In the above description, the term layer wasused to also cover the concept of views simultaneously.

The above embodiments described a scalable video decoder that can startdecoding of an enhancement layer picture before the decoding of theassociated base layer picture has been finalized. Picture area decodingis pipelined based on high level parallelization tools that are used ineach layer. Base and enhancement layer decoder can operate in parallelto each other and also in parallel within the actual layer. The amountof parallelism within each layer can differ between base and enhancementlayers. Further, a signaling was described that is necessitated toproperly set up a parallel decoding environment for a specificbitstream.

As a general note the following is noted: the above embodimentsdescribed decoders and correspondingly designed encoders in accordancewith various aspects. As far as all these aspects are concerned, it isnoted that they may all have in common that decoder and encoder supportWPP and/or tile parallel processing and accordingly wherever detailsthereabout are described, these details shall be treated as concurrentlyapplying to any of the other aspects and their correspond descriptionsso as to either lead to new embodiments of these other aspects, orcomplement the descriptions of embodiments of these otheraspects—irrespective as to whether the respective aspect onto which arespective description is be transferred, has been described using terms“portions”, “spatial segments” or the like instead of tiles/substreamsfor more generally denoting parallel processable segments of a picture.The same applies to details regarding coding/prediction parameters anddescriptions setting out possible ways of subdividing the pictures: allaspects may be implemented so as to result in decoders/encoders whichuse a subdivisioning into LCUs/CTBs with determining tiles and/orsusbtreams in units of the same. Additionally, in any of these aspects,the LCUs/CTBs may further be subdivided into coding blocks by use ofrecursive multi-tree subdivision as described above with respect to asubset of the aspects and their embodiments. And additionally oralternatively, the slice concept may be adopted by embodiments of allaspects, from those aspects with respect to which the relation betweenslices and substreams/tiles has been described.

Although some aspects have been described in the context of anapparatus, it is clear that these aspects also represent a descriptionof the corresponding method, where a block or device corresponds to amethod step or a feature of a method step. Analogously, aspectsdescribed in the context of a method step also represent a descriptionof a corresponding block or item or feature of a correspondingapparatus. Some or all of the method steps may be executed by (or using)a hardware apparatus, like for example, a microprocessor, a programmablecomputer or an electronic circuit. In some embodiments, some one or moreof the most important method steps may be executed by such an apparatus.

Depending on certain implementation requirements, embodiments of theinvention can be implemented in hardware or in software. Theimplementation can be performed using a digital storage medium, forexample a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM,an EEPROM or a FLASH memory, having electronically readable controlsignals stored thereon, which cooperate (or are capable of cooperating)with a programmable computer system such that the respective method isperformed. Therefore, the digital storage medium may be computerreadable.

Some embodiments according to the invention comprise a data carrierhaving electronically readable control signals, which are capable ofcooperating with a programmable computer system, such that one of themethods described herein is performed.

Generally, embodiments of the present invention can be implemented as acomputer program product with a program code, the program code beingoperative for performing one of the methods when the computer programproduct runs on a computer. The program code may for example be storedon a machine readable carrier.

Other embodiments comprise the computer program for performing one ofthe methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the inventive method is, therefore, acomputer program having a program code for performing one of the methodsdescribed herein, when the computer program runs on a computer.

A further embodiment of the inventive methods is, therefore, a datacarrier (or a digital storage medium, or a computer-readable medium)comprising, recorded thereon, the computer program for performing one ofthe methods described herein. The data carrier, the digital storagemedium or the recorded medium are typically tangible and/ornon-transitionary.

A further embodiment of the inventive method is, therefore, a datastream or a sequence of signals representing the computer program forperforming one of the methods described herein. The data stream or thesequence of signals may for example be configured to be transferred viaa data communication connection, for example via the Internet.

A further embodiment comprises a processing means, for example acomputer, or a programmable logic device, configured to or adapted toperform one of the methods described herein.

A further embodiment comprises a computer having installed thereon thecomputer program for performing one of the methods described herein.

A further embodiment according to the invention comprises an apparatusor a system configured to transfer (for example, electronically oroptically) a computer program for performing one of the methodsdescribed herein to a receiver. The receiver may, for example, be acomputer, a mobile device, a memory device or the like. The apparatus orsystem may, for example, comprise a file server for transferring thecomputer program to the receiver.

In some embodiments, a programmable logic device (for example a fieldprogrammable gate array) may be used to perform some or all of thefunctionalities of the methods described herein. In some embodiments, afield programmable gate array may cooperate with a microprocessor inorder to perform one of the methods described herein. Generally, themethods are performed by any hardware apparatus.

The apparatus described herein may be implemented using a hardwareapparatus, or using a computer, or using a combination of a hardwareapparatus and a computer.

The methods described herein may be performed using a hardwareapparatus, or using a computer, or using a combination of a hardwareapparatus and a computer.

The above described embodiments are merely illustrative for theprinciples of the present invention. It is understood that modificationsand variations of the arrangements and the details described herein willbe apparent to others skilled in the art. It is the intent, therefore,to be limited only by the scope of the impending patent claims and notby the specific details presented by way of description and explanationof the embodiments herein.

Thus, the following aspects have been discussed above:

A first aspect concerns a video decoder for decoding a multi-layer videodata stream 40 into which a scene is coded in a hierarchy of layersusing inter-layer prediction from a first layer to a second layer, thevideo decoder supporting parallel decoding the multi-layer video datastream in spatial segments 80 into which pictures 12, 15 of the layersare sub-divided, wherein the decoder is configured to

inspect a long-term syntax element structure (606; e.g.tile_boundaries_aligned_flag) of the multi-layer video data stream so asto

interpret the long-term syntax element structure assuming a value out ofa first possible-values-set (e.g. tile_boundaries_aligned_flag=1), as aguarantee that, during a predetermined time period 608, the pictures 15of the second layer are subdivided so that borders between the spatialsegments of the pictures of the second layer overlay every border of thespatial segments of the pictures 12 of the first layer and periodicallydetermine, in time intervals 604 smaller than the predetermined timeperiod, the subdivision of the pictures of the first layer and thesecond layer into the spatial segments based on short-term syntaxelements (602; e.g. column_width_minus1[i] and column_width_minus1[i])of the multi-layer video data stream, and

if the long-term syntax element structure has assumed a value out of asecond possible-values-set (e.g. tile_boundaries_aligned_flag=0),periodically determine, in the time intervals smaller than thepredetermined time period, the subdivision of the pictures of the layersinto the spatial segments from the short-term syntax elements of themulti-layer video data stream such that, at least for a first possiblevalue of the short-term syntax elements, there exists a border betweenthe spatial segments of the pictures of the second layer not overlayingany of the borders of the spatial segments of the first layer, and, atleast for a second possible value of the short-term syntax elements,borders between the spatial segments of the pictures of the second layeroverlay every border of the spatial segments of the first layer.

A second aspect concerns the video decoder according to aspect 1,wherein the video decoder is configured to

decode the pictures of the layers using intra-picture spatial predictionwith disrupting the intra-picture spatial prediction for each spatialsegment at a boundary of the respective spatial segment; or

decode the pictures of the layers using intra-picture spatial predictionby decoding the spatial segments of a picture of the first layer inparallel with supporting the intra-picture spatial prediction crossingboundaries of the spatial segments of the picture of the first layer andobeying a decoding delay between the decoding of the spatial segments ofthe picture of the first layer, and by decoding the spatial segments ofa picture of the second layer in parallel with supporting theintra-picture spatial prediction crossing boundaries of the spatialsegments of the picture of the second layer and obeying a decoding delaybetween the decoding of the spatial segments of the picture of thesecond layer.

A third aspect concerns the video decoder according to aspect 1 or 2,supporting tile parallel decoding the multi-layer video data stream intiles into which pictures of the layers are sub-divided, wherein thedecoder is configured to

interpret the long-term syntax element assuming the value out of thefirst possible-values-set, as a guarantee that, during a predeterminedtime period, the pictures of the second layer are subdivided so thatborders between the tiles of the pictures of the second layer overlayevery border of the tiles of the first layer and periodically determine,in time intervals smaller than the predetermined time period, a tilerefinement of a subdivision of the pictures of the second layer relativeto the first layer based on the short-term syntax elements, and

if the long-term syntax element has assumed the value out of the secondpossible-values-set, periodically determine, in time intervals smallerthan the predetermined time period, the subdivision of the pictures ofthe layers into the tiles from the short-term syntax elements of themulti-layer video data stream such that, at least for a first possiblevalue of the short-term syntax elements, there exists a border betweenthe tiles of the pictures of the second layer not overlaying any of theborders of the tiles of the first layer, and, at least for a secondpossible value of the short-term syntax elements, borders between thetiles of the pictures of the second layer overlay every border of thetiles of the first layer.

A fourth aspect concerns the video decoder according to aspect 3,wherein the video decoder is configured to

decode the pictures of the layers using intra-picture spatial predictionwith disrupting the intra-picture spatial prediction for each tile at aboundary of the respective tile.

A fifth aspect concerns the video decoder according to aspect 1 or 2,wherein the decoder is configured to

interpret the long-term syntax element structure assuming a value out ofa first possible-values-set, as a guarantee that, during a predeterminedtime period, the pictures of the second layer are subdivided so thateach spatial segment of the pictures of the first layer is made up ofexactly n spatial segments of the pictures of second layer with ndepending on the value of the long-term syntax element structure, and

if the long-term syntax element is set to a value of a secondpossible-values-set, periodically determine, in time intervals smallerthan the predetermined time period, the inter-layer offset within thepredetermined period based on the short-term syntax elements of the ofthe multi-layer video data stream.

A sixth aspect concerns the video decoder according to aspect 1 or 2,wherein the decoder is configured to decide on commencing or notcommencing a trial to decode the second layer of the multi-layer videodata stream depending on the long-term syntax element structure assumingthe value out of the first possible-values or not.

A seventh aspect concerns the video decoder according to any of aspects1 to 6, wherein the video decoder is a hybrid video decoder.

An eighth aspect concerns a video encoder for encoding a scene into amulti-layer video data stream in a hierarchy of layers using inter-layerprediction from a first layer to a second layer so that the multi-layervideo data stream is decodable in parallel in spatial segments intowhich pictures of the layers are sub-divided, wherein the encoder isconfigured to

insert a long-term syntax element structure 606 and short-term syntaxelements 602 into the multi-layer video data stream, the short-termsyntax elements defining, in time intervals, the subdivision of thepictures of the first layer and the second layer into the spatialsegments, and

switch between setting the long-term syntax element structure to

-   -   a value out of a first possible-values-set with, during a        predetermined time period 608 greater than the time intervals,        setting the short-term syntax elements to a proper subset out of        a set of possible settings, the proper subset being chosen so        that, during the predetermined time period, the pictures of the        second layer are subdivided so that borders between the spatial        segments of the pictures of the second layer overlay every        border of the spatial segments of the first layer, or    -   a value out of a second possible-values-set with, during the        predetermined time period, setting the short-term syntax        elements to any of the set of possible settings, the set of        possible settings encompassing at least one setting according to        which a border exists between the spatial segments of the        pictures of the second layer not overlaying any of the borders        of the spatial segments of the first layer, and at least another        setting according to which borders between the spatial segments        of the pictures of the second layer overlay every border of the        spatial segments of the first layer.

A ninth aspect concerns the video encoder according to aspect 8, whereinthe video encoder is configured to

encode the pictures of the layers using intra-picture spatial predictionwith disrupting the intra-picture spatial prediction for each spatialsegment at a boundary of the respective spatial segment; or

encode the pictures of the layers using intra-picture spatial predictionand entropy encoding with adapting entropy context probabilities, bysupporting the intra-picture spatial prediction crossing boundaries ofthe spatial segments of a picture of the first layer and initializingthe entropy context probabilities for the entropy encoding of thesubsets of the spatial segments of the picture of the first layerindividually or under adopting the entropy context probabilities of a,in accordance with an order among the subsets, previous subset of thespatial segments of the picture of the first layer at an intermediatelyadapted state, and by supporting the intra-picture spatial predictioncrossing boundaries of the spatial segments of a picture of the secondlayer and initializing the entropy context probabilities for the entropyencoding of the subsets of the spatial segments of the picture of thesecond layer individually or under adopting the entropy contextprobabilities of a, in accordance with an order among the subsets,previous subset of the spatial segments of the picture of the secondlayer at an intermediately adapted state.

A tenth aspect concerns the video encoder according to aspect 8 or 9,wherein the spatial segments are tiles and the encoder is configured to

if setting the long-term syntax element structure to

-   -   a value out of the first possible-values-set        (tile_boundaries_aligned_flag=1), set, during a predetermined        time period greater than the time intervals, the short-term        syntax elements to a proper subset out of a set of possible        settings, the proper subset being chosen so that, during the        predetermined time period, the second layer's pictures'        subdivision into tiles coincides or refines the first layer's        pictures' subdivision into tiles, or    -   a value out of a second possible-values-set        (tile_boundaries_aligned_flag=0), set, during the predetermined        time period, the short-term syntax elements to any of the set of        possible settings such that the short-term syntax elements are,        at least for one time interval during the predetermined time        period, set to a first possible value of the set of possible        settings, according to which there exists a border between the        tiles of the pictures of the second layer not overlaying any of        the borders of the tiles of the first layer, and, at least for        another time interval during the predetermined time period, set        to a second possible value of the set of possible settings,        according to which borders between the tiles of the pictures of        the second layer overlay every border of the tiles of the first        layer.

An eleventh aspect concerns the video encoder according to aspect 10,wherein the video encoder is configured to

encode the pictures of the layers using intra-picture spatial predictionwith disrupting the intra-picture spatial prediction for each tile at aboundary of the respective tile.

A twelfth aspect concerns the video encoder according to aspect 8 or 9,wherein the encoder is configured to

if setting the long-term syntax element structure to

-   -   a value out of the first possible-values-set, setting using the        predetermined time period greater than the time intervals, the        short-term syntax elements to a proper subset out of a set of        possible settings, the proper subset being chosen so that,        during the predetermined time period, each spatial segment of        the pictures of the first layer is made up of exactly n spatial        segments of the pictures of second layer with n depending on the        value of the long-term syntax element structure.

A thirteenth aspect concerns a decoder for decoding a spatially scalablebitstream 40 into which a picture is coded in different spatial layersand, for at least one of the spatial layers, in first spatial segments,wherein the decoder is configured to

upsample a picture 12 of a first spatial layer so as to obtain anupsampled reference picture and predict a picture 15 of a second spatiallayer using the upsampled reference picture, wherein the decoder isresponsive to a syntax element (616; e.g.,independent_tile_upsampling_idc) in the spatially scalable bitstream soas to, depending on the syntax element, interpolate 620 the picture ofthe first spatial layer

such that any partition of a partitioning 622 of the upsampled referencepicture, which depends on the first spatial segments, is independentfrom portions of the picture of the first spatial layer covered by anyof the other partitions of the partitioning, or

such that any partition of the partitioning 622 of the upsampledreference picture is dependent on portions of the picture of the firstspatial layer covered by another partition of the partitioning spatiallyneighbouring the respective partition.

A fourteenth aspect concerns the decoder according to aspect 13, whereinthe decoder is configured to decode the different spatial layers inparallel.

A fifteenth aspect concerns the decoder according to aspect 13 or 14,wherein the decoder is responsive to the syntax element 616 in thespatially scalable bitstream so as to, depending on the syntax element,interpolate 620 the picture of the first spatial layer such that anyportion of the upsampled reference picture spatially covered by_any ofthe first spatial segments, is independent from portions of the pictureof the first spatial layer covered by any other of the first spatialsegments, or such that any portion of the upsampled reference picturespatially covered by any of the first spatial segments, is dependent onportions of the picture of the first spatial layer covered by any otherspatial segment of the first spatial segments, neighbouring therespective spatial segment.

A sixteenth aspect concerns the decoder according to any of aspects 13to 15, wherein the spatially scalable bitstream has the picture of thesecond spatial layer coded thereinto in the first spatial segments.

A seventeenth aspect concerns the decoder according to any of aspects 13to 16, wherein the decoder is configured to perform the decoding usingintra-picture spatial prediction

with disrupting the intra-picture spatial prediction for each firstspatial segment at a boundary of the respective first spatial segment or

with supporting the intra-picture spatial prediction crossing boundariesof the first spatial segments, entropy decoding each first spatialsegment using an adaptation of entropy context probabilities andinitializing the entropy context probabilities of the first spatialsegments independent from any other first spatial segment or underadopting the entropy context probabilities of a, in accordance with anorder among the first spatial segments, previous first spatial segmentas adapted up to an intermediate position of the previous first spatialsegment.

An eighteenth aspect concerns the decoder according to aspect 13 or 14,wherein the spatially scalable bitstream has the picture of the firstspatial layer coded thereinto in the first spatial segments, wherein thespatially scalable bitstream has the picture of the second spatial layercoded thereinto in second spatial segments, wherein the decoder isresponsive to the syntax element 606 in the spatially scalable bitstreamso as to, depending on the syntax element, interpolate the picture ofthe first spatial layer (e.g., independent_tile_upsampling_idc=2) suchthat any portion of the upsampled reference picture spatially covered byany of the second tiles, is independent from portions of the picture ofthe first spatial layer spatially covered by any other of the secondtiles, or

(e.g., independent_tile_upsampling_idc=1) such that any partition of theupsampled reference picture spatially confined by spatially co-locatedboundaries of the first and second tiles is independent from portions ofthe picture of the first spatial layer covered by any other partition ofthe upsampled reference picture, spatially confined by spatiallyco-located boundaries of the first and second tiles, or

(e.g., independent_tile_upsampling_idc=0) such that any portion of theupsampled reference picture spatially covered by any of the secondtiles, is dependent on portions of the picture of the first spatiallayer covered by any other second tile neighbouring the respective tile.

A nineteenth aspect concerns the decoder according to any of aspects 13to 18, wherein the decoder is configured to, in order to achieve theindependence, fill fragments of a filter kernel used in theinterpolation of the picture of the first spatial layer, which protrudeinto any such portion of the picture of the first spatial layer, coveredby any of the other partitions of the partitioning, using a fallbackrule according to which the fragments are filled independent from therespective portion of the picture of the first spatial layer into whichthe filter kernel protrudes.

A twentieth aspect concerns the decoder according to aspect 19, whereinthe decoder is configured to use the fallback rule also in fillingfragments of the filter kernel protruding an outer boundary of thepicture of the first spatial layer.

A twenty-first aspect concerns the decoder according to any of aspects13 to 20, wherein the decoder is a video decoder and configured to byresponsive to the syntax element 606 in the spatially scalable bitstreamon a per-picture or per-picture-sequence basis.

A twenty-second aspect concerns the decoder according to any of aspects13 to 21, wherein the spatially scalable bitstream has the picture ofthe first spatial layer coded thereinto in the first spatial segments,wherein the spatially scalable bitstream has the picture of the secondspatial layer coded thereinto in second spatial segments, wherein thepartitioning's borders either correspond to a logical AND of a spatialoverlay of borders of the first and second spatial segments or theborders of the second spatial segments, wherein the decoder isresponsive to the syntax element 606 in the spatially scalable bitstreamso as to, depending on the syntax element, fill fragments of a filterkernel used in the interpolation of the picture of the first spatiallayer, which protrude from one partition into a neighboring partition ofthe partitioning using a fallback rule according to which the fragmentsare filled independent from the respective portion of the picture of thefirst spatial layer into which the filter kernel protrudes, or using therespective portion of the picture of the first spatial layer into whichthe filter kernel protrudes.

A twenty-third aspect concerns the decoder according to any of aspects13 to 22, wherein the decoder is configured to decode the first andsecond layers in parallel using an inter-layer offset being dependent onthe syntax element 606.

A twenty-fourth aspect concerns the decoder according to aspect 13 or23, wherein the decoder is configured to place the partitioning'sborders so as to correspond to the logical AND of a spatial overlay ofborders of the first and second spatial segments or the borders of thesecond spatial segments, depending on the syntax element.

A twenty-fifth aspect concerns an encoder for encoding a picture into aspatially scalable bitstream in different spatial layers and, for atleast one of the spatial layers, in first spatial segments, wherein theencoder is configured to

upsample a picture of a first spatial layer so as to obtain an upsampledreference picture and predict a picture of a second spatial layer usingthe upsampled reference picture, wherein the encoder is configured toset and insert a syntax element 606 into the spatially scalablebitstream and, depending on the syntax element, interpolate the pictureof the first spatial layer

such that any partition of a partitioning of the upsampled referencepicture, which depends on the first spatial segments, is independentfrom portions of the picture of the first spatial layer covered by anyof the other partitions of the partitioning, or

such that any partition of the partitioning of the upsampled referencepicture is dependent on portions of the picture of the first spatiallayer covered by another partition of the partitioning spatiallyneighbouring the respective partition.

A twenty-sixth aspect concerns the encoder according to aspect 25,wherein the encoder is configured to set and insert the syntax elementinto the spatially scalable bitstream and to, depending on the syntaxelement, interpolate the picture of the first spatial layer

such that any portion of the upsampled reference picture spatiallycovered by any of the first spatial segments, is independent fromportions of the picture of the first spatial layer covered by any otherof the first spatial segments, or

such that any portion of the upsampled reference picture spatiallycovered by any of the first spatial segments, is dependent on portionsof the picture of the first spatial layer covered by any other spatialsegment of the first spatial segments, neighbouring the respective firstspatial segment.

A twenty-seventh aspect concerns the encoder according to aspect 25 or26, wherein the encoder is configured to encode the picture of the firstspatial layer thereinto the spatially scalable bitstream in the firstspatial segments.

A twenty-eighth aspect concerns the encoder according to aspect 27,wherein the encoder is configured to encode the picture of the firstspatial layers using intra-picture spatial prediction with disruptingthe intra-picture spatial prediction for each first spatial segment at aboundary of the respective first spatial segment.

A twenty-ninth aspect concerns the encoder according to aspect 27 or 28,wherein the encoder is configured to the picture of the second spatiallayer coded into the spatially scalable bitstream in second spatialsegments, wherein the encoder is configured to set and insert the syntaxelement into the spatially scalable bitstream and, depending on thesyntax element, interpolate the picture of the first spatial layer

such that any portion of the upsampled reference picture spatiallycovered by any of the second spatial segments, is independent fromportions of the picture of the first spatial layer spatially covered byany other of the second spatial segments, or

such that any partition of the upsampled reference picture spatiallyconfined by spatially co-located boundaries of the first and secondspatial segments is independent from portions of the picture of thefirst spatial layer covered by any other partition of the upsampledreference picture, spatially confined by spatially co-located boundariesof the first and second spatial segments, or

such that any portion of the upsampled reference picture spatiallycovered by any of the second spatial segments, is dependent on portionsof the picture of the first spatial layer covered by any other secondspatial segment neighbouring the respective spatial segment.

A thirtieth aspect concerns the encoder according to aspect 29, whereinthe encoder is configured to encode the picture of the second spatiallayer using intra-picture spatial prediction with disrupting theintra-picture spatial prediction for each second spatial segment at aboundary of the respective second spatial segment.

A thirty-first aspect concerns the encoder according to any of aspects25 to 30, wherein the encoder is configured to, in order to achieve theindependence from any portion of the picture of the first spatial layer,fill fragments of a filter kernel used in the interpolation of thepicture of the first spatial layer, protruding into any such portion ofthe picture of the first spatial layer, using a fallback rule accordingto which the fragments are filled independent from the respectiveportion of the picture of the first spatial layer into which the filterkernel protrudes.

A thirty-second aspect concerns the encoder according to aspect 31,wherein the encoder is configured to use the fallback rule also infilling fragments of the filter kernel protruding an outer boundary ofthe picture of the first spatial layer.

A thirty-third aspect concerns the encoder according to any of aspects25 to 32, wherein the encoder is a video encoder and configured to setand insert the syntax element into the spatially scalable bitstream on aper-picture or per-picture-sequence basis.

A thirty-fourth aspect concerns a video decoder for decoding amulti-layer video data stream into which a scene is coded in a hierarchyof layers using inter-layer prediction from a first layer to a secondlayer, the video decoder supporting parallel decoding the multi-layervideo data stream in spatial segments into which pictures of the layersare partitioned by sequentially traversing the spatial segments in atemporally overlapping manner with an inter-layer delay between thetraversal of spatial segments of pictures of the first layer relative tothe traversal of spatial segments of pictures of the second layer, thevideo decoder configured to

inspect a long-term syntax element structure (e.g.min_spatial_segment_delay) of the multi-layer video data stream so as to

if the long-term syntax element structure (e.g.min_spatial_segment_delay; ≠0) is set to a value of a first set ofpossible values, use the value of the long-term syntax element todetermine the inter-layer offset in advance for a predetermined timeperiod and periodically determine, in time intervals smaller than thepredetermined time period, sizes and locations of the spatial segmentsof the pictures of the first layer and the spatial segments of thepictures of the second layer, and spatial sampling resolutions of thepictures of the first layer and the pictures of the second layer,respectively, based on short-term syntax elements 602 of the of themulti-layer video data stream;

if the long-term syntax element is set to a value of a second set (e.g.min_spatial_segment_delay=0) of possible values disjoint to the firstset of possible values, periodically determine, in time intervalssmaller than the predetermined time period, the inter-layer offsetwithin the predetermined period based on the short-term syntax elementsof the of the multi-layer video data stream.

A thirty-fifth aspect concerns the video decoder of aspect 34, whereinthe video decoder is configured to, in parallel, decode the multi-layervideo data stream in sub-streams out of which pictures of the layers arepartitioned and which are composed of rows of blocks into which thepictures of the layers are regularly sub-divided, using wavefrontparallel processing by sequentially traversing the substreams in atemporally overlapping manner with an intra-picture inter-substreamdelay between the traversal of immediately consecutive substreams of thesame picture and the inter-layer offset between the traversal ofsubstreams of pictures of the first layer relative to the traversal ofsubstreams of pictures of the second layer.

A thirty-sixth aspect concerns the video decoder according to aspect 35,wherein the video decoder is configured to

decode the sub-streams in parallel with supporting intra-picture spatialprediction crossing boundaries of the sub-streams.

A thirty-seventh aspect concerns the video decoder of aspect 34, whereinthe video decoder is configured to decode the multi-layer video datastream in tiles out of which pictures of the layers are partitioned,with traversing the tiles at a tile order among the tiles within each ofthe pictures of the first and second layers and decoding immediatelysucceeding tiles of pictures of the first layer and immediatelysucceeding tiles of pictures of the second layer in parallel with theinter-layer offset between the traversal of the tiles of the pictures ofthe first layer relative to the traversal of tiles of the pictures ofthe second layer.

A thirty-eighth aspect concerns the video decoder according to aspect37, wherein the video decoder is configured to

decode the pictures of the first and second layers using intra-picturespatial prediction with disrupting the intra-picture spatial predictionfor each tile at a boundary of the respective tile.

A thirty-ninth aspect concerns the video decoder of any of aspects 34 to38, wherein the video decoder is configured to use the value of thelong-term syntax element in determining the inter-layer offset by usingthe value of the long-term syntax element as a measure for theinter-layer offset in units of spatial segments of the pictures of thefirst layer.

A fortieth aspect concerns the video decoder of any of aspects 34 to 39,wherein the video decoder is configured to use the value of thelong-term syntax element in determining the inter-layer offset by usingthe value of the long-term syntax element as a number of spatialsegments of the pictures of the first layer by which the decoding of afirst spatial segment of the pictures of the second layer is to bedelayed relative to commencing the decoding and traversal of thepictures of the first layer.

A forty-first aspect concerns a video encoder for encoding a scene intoa multi-layer video data stream in a hierarchy of layers usinginter-layer prediction from a first layer to a second layer such thatthe multi-layer video data stream is decodable in spatial segments intowhich pictures of the layers are partitioned by sequentially traversingthe spatial segments in a temporally overlapping manner with aninter-layer offset between the traversal of spatial segments of picturesof the first layer relative to the traversal of spatial segments ofpictures of the second layer, the video encoder configured to

Insert and set a long-term syntax element structure(min_spatial_segment_delay) and short-term syntax elements into themulti-layer video data stream, the short-term syntax elements of the ofthe multi-layer video data stream defining, in a periodical basis, intime intervals, sizes and locations of the spatial segments of thepictures of the first layer and the spatial segments of the pictures ofthe second layer, and spatial sampling resolutions of the pictures ofthe first layer and the pictures of the second layer, respectively,

wherein the encoder is configured to switch between setting

the long-term syntax element structure (min_spatial_segment_delay≠0) toa value of a first set of possible values, with the value signaling theinter-layer offset for a predetermined time period greater than the timeintervals, with, for the predetermined time period, setting theshort-term syntax elements to a proper subset out of a set of possiblesettings, the proper subset being chosen so that, during thepredetermined time period, the sizes and the locations of the spatialsegments of the pictures of the first layer and the spatial segments ofthe pictures of the second layer, and the spatial sampling resolutionsof the pictures of the first layer and the pictures of the second layer,respectively, enable decoding the multi-layer video data stream bysequentially traversing the spatial segments in the temporallyoverlapping manner with an actual inter-layer offset which is smallerthan or equal to the inter-layer offset signaled by the long-term syntaxelement, between the traversal of spatial segments of pictures of thefirst layer relative to the traversal of spatial segments of pictures ofthe second layer;

the long-term syntax element to a value of a second set(min_spatial_segment_delay=0) of possible values disjoint to the firstset of possible values, with, during the predetermined time period,setting the short-term syntax elements to any of the set of possiblesettings, the set of possible settings encompassing at least one settingaccording to which the sizes and the locations of the spatial segmentsof the pictures of the first layer and the spatial segments of thepictures of the second layer, and the spatial sampling resolutions ofthe pictures of the first layer and the pictures of the second layer,respectively, disable decoding the multi-layer video data stream bysequentially traversing the spatial segments in the temporallyoverlapping manner with an actual inter-layer offset which is smallerthan or equal to the inter-layer offset signaled by the long-term syntaxelement, between the traversal of spatial segments of pictures of thefirst layer relative to the traversal of spatial segments of pictures ofthe second layer, and at least another setting according to which thesizes and the locations of the spatial segments of the pictures of thefirst layer and the spatial segments of the pictures of the secondlayer, and the spatial sampling resolutions of the pictures of the firstlayer and the pictures of the second layer, respectively, enabledecoding the multi-layer video data stream by sequentially traversingthe spatial segments in the temporally overlapping manner with an actualinter-layer offset which is smaller than or equal to the inter-layeroffset signaled by the long-term syntax element, between the traversalof spatial segments of pictures of the first layer relative to thetraversal of spatial segments of pictures of the second layer.

A forty-second aspect concerns the video encoder of aspect 41, whereinthe video encoder is configured perform the encoding such that thespatial segments are substreams out of which pictures of the layers arepartitioned and which are composed of rows of blocks into which thepictures of the layers are regularly sub-divided, in a manner allowing,in parallel, decoding the multi-layer video data stream in thesub-streams using wavefront parallel processing by sequentiallytraversing the substreams in a temporally overlapping manner with anintra-picture inter-substream delay between the traversal of immediatelyconsecutive substreams of the same picture and the inter-layer offsetbetween the traversal of substreams of pictures of the first layerrelative to the traversal of substreams of pictures of the second layer.

A forty-third aspect concerns the video encoder according to aspect 42,wherein the video encoder is configured to

encode the sub-streams using intra-picture spatial prediction andentropy encoding with adapting entropy context probabilities, bysupporting the intra-picture spatial prediction crossing boundaries ofthe sub-streams and initializing the entropy context probabilities forentropy encoding the substreams individually or under adopting theentropy context probabilities of a, in accordance with an order amongthe substreams, previous substream at an intermediately adapted state.

A forty-fourth aspect concerns the video encoder of aspect 41, whereinthe video encoder is configured perform the encoding such that thespatial segments are tiles out of which pictures of the layers arepartitioned, in a manner allowing decoding the multi-layer video datastream in the tiles by traversing the tiles at a tile order among thetiles within each of the pictures of the first and second layers anddecoding immediately succeeding tiles of pictures of the first layer andimmediately succeeding tiles of pictures of the second layer in parallelwith the inter-layer offset between the traversal of the tiles of thepictures of the first layer relative to the traversal of tiles of thepictures of the second layer.

A forty-fifth aspect concerns the video encoder according to aspect 44,wherein the video encoder is configured to

encode the pictures of the first and second layers using intra-picturespatial prediction with disrupting the intra-picture spatial predictionfor each tile at a boundary of the respective tile.

A forty-sixth aspect concerns the video encoder of any of aspects 41 to45, wherein the video encoder is configured such that the value of thelong-term syntax element defines a measure for the inter-layer offset inunits of spatial segments of the pictures of the first layer.

A forty-seventh aspect concerns the video decoder of any of aspects 41to 46, wherein the video encoder is configured to set the value of thelong-term syntax element so as to signal a number of spatial segments ofthe pictures of the first layer by which the decoding of a first spatialsegment of the pictures of the second layer is to be delayed relative tocommencing the decoding and traversal of the pictures of the firstlayer.

A forty-eighth aspect concerns a network entity for processing amulti-layer video data stream into which a scene is coded in layers sothat, in each layer, the scene is coded in a different operational pointof a scalability space spanned by scalability dimensions, wherein themulti-layer video data stream is composed of first NAL units each ofwhich as associated with one of the layers, and second NAL unitsinterspersed within the first NAL units and presenting generalinformation on the multi-layer video data stream, the network entitybeing configured to

inspect a type indicator field (696, e.g.dedicated_scalability_ids_flag) in the second NAL units;

if the type indicator field has a first state (e.g.dedicated_scalability_ids_flag=0), read mapping information (e.g.layer_id_in_nuh[i], dimension_id[i][j]) mapping possible values of alayer indicator field (e.g. layer_id) in the first NAL unit headers tothe operational points from the second NAL units and associate the firstNAL units with the operational points in the first NAL units via thelayer indicator field and the mapping information;

if the type indicator field has a second state(dedicated_scalability_ids_flag=1), associate the first NAL units withthe operational points by splitting the layer indicator field in thefirst NAL units into more than one portion and locating the operationalpoint of the first NAL units by using the values of the portions ascoordinates of a vector within the scalability space.

A forty-ninth aspect concerns the network entity according to aspect 48,wherein the network entity is configured to

if the type indicator field has the second state(dedicated_scalability_ids_flag=1), associate the first NAL units withthe operational points by dividing the layer indicator field in thefirst NAL units into the more than one portion according to a syntaxelement (dim_ension_id_len_minus1) in the second NAL units, locating theoperational point of the first NAL units by using the values of theportions as coordinates of a vector within the scalability space, andsemantically determining the scalability dimensions according to afurther syntax element (scalability_mask) in the second NAL units.

A fiftieth aspect concerns the network entity according to aspect 48 or49, wherein the network entity is configured to

if the type indicator field has the first state(dedicated_scalability_ids_flag=0), determining number p and semanticmeaning of the scalability dimensions from a further syntax element(scalability_mask) in the second NAL units and associate the possiblevalues of the layer indicator field with the operational points byreading a list 708 of p-dimensional vectors from the second NAL units.

A fifty-first aspect concerns the network entity according to aspect 50,wherein the network entity is configured to skip reading the list fromthe second NAL units if the type indicator field has the second state.

A fifty-second aspect concerns the network entity according to any ofaspects 49 or 51, wherein the network entity is configured to read thefurther syntax element from the second NAL units irrespective of thetype indicator field having the first or the second state, and such thatthe size of the layer indicator field is the same irrespective of thetype indicator field having the first or the second state.

A fifty-third aspect concerns the network entity according to any ofaspects 48 or 52, wherein the network entity comprises a video decoder.

A fifty-fourth aspect concerns a video encoder for encoding a scene intoa multi-layer video data stream in layers so that, in each layer, thescene is coded in a different operational point of a scalability spacespanned by scalability dimensions, wherein the multi-layer video datastream is composed of first NAL units each of which as associated withone of the layers, and second NAL units interspersed within the firstNAL units and presenting general information on the multi-layer videodata stream, the video encoder being configured to

insert a type indicator field into the second NAL units and switchbetween setting

the type indicator field such that same has a first state, withinserting mapping information mapping possible values of a layerindicator field in the first NAL unit headers to the operational pointsinto the second NAL units and setting the layer indicator filed in thefirst NAL units such that the first NAL units' operational points areassociated with the respective layer indicator field via the mappinginformation;

the type indicator field such that same has a second state(dedicated_scalability_ids_flag=1), with setting the layer indicatorfield in the first NAL units by splitting the layer indicator field inthe first NAL units into more than one portion and setting the more thanportion so that the values of the portions correspond to coordinates ofa vector within the scalability space, pointing to the operational pointassociated with the respective first NAL unit.

A fifty-fifth aspect concerns the video encoder according to aspect 54,wherein the video encoder is configured to

when setting the type indicator field such that same has the secondstate, set and insert a syntax element into the second NAL units whichdefines as to who the layer indicator field in the first NAL units aredivided into the more than one portion, and set and insert a furthersyntax element into the second NAL units which semantically defines thescalability dimensions.

A fifty-sixth aspect concerns a multi-layer video datastream into whicha scene is coded in layers so that, in each layer, the scene is coded ina different operational point of a scalability space spanned byscalability dimensions, wherein the multi-layer video data stream iscomposed of first NAL units each of which as associated with one of thelayers, and second NAL units interspersed within the first NAL units andpresenting general information on the multi-layer video data stream,wherein a type indicator field (696, e.g.dedicated_scalability_ids_flag) is present in the second NAL unitsdepending on which

if the type indicator field has a first state (e.g.dedicated_scalability_ids_flag=0), mapping information in the second NALunits maps possible values of a layer indicator field (e.g. layer_id) inthe first NAL unit headers to the operational points;

if the type indicator field has a second state(dedicated_scalability_ids_flag=1), the layer indicator field in thefirst NAL units is split into more than one portion with the operationalpoint of the first NAL units being defined by the values of the portionsas coordinates of a vector within the scalability space.

A fifty-seventh aspect concerns a transport layer decoder for subjectinga multi-layer video data stream into which a scene is coded in layers,to decoding by a multi-standard multi-layer decoder, wherein themulti-layer video data stream is composed of NAL units each of which isassociated with one of the layers, wherein the layers are associatedwith different codecs so that, for each layer, the NAL units associatedwith the respective layer are coded using the codec with which therespective layer is associated, the transport layer decoder beingconfigured to

identify, for each NAL unit, as to which codec same is associated with,and

hand over the NAL units of the multi-layer video data stream to themulti-standard multi-layer decoder which uses inter-layer predictionbetween layers associated with different codecs to decode themulti-layer video data stream.

A fifty-eighth aspect concerns the video decoder according to aspect 57,further configured to

encapsulate NAL units having been identified as being associated with alayer which is associated with any codec different from a predeterminedcodec, using NAL unit headers of the predetermined codec having a NALunit type indicator set to a state indicating the codec of therespective layer.

A fifty-ninth aspect concerns the video decoder according to aspect 57or 58, further configured to

perform the identification depending on a channel on which the NAL unitsarrive respectively.

A sixtieth aspect concerns the video decoder according to any of aspects57 or 59, further configured to

perform the handing over such that the NAL units associated withdifferent codecs are handed over to the multi-standard multi-layerdecoder on different channels.

A sixty-first aspect concerns the video decoder according to aspect 57or 60, further configured to

provide each NAL unit with meta data indicating the codec with which thelayer associated with the respective NAL unit is associated.

A sixty-second aspect concerns a video decoder for decoding amulti-layer video data stream into which a scene is coded in a hierarchyof layers using inter-layer prediction from portions of a first layer toco-located portions of a second layer, wherein pictures of the firstlayer are sub-divided into an array of first blocks and pictures of thesecond layer are sub-divided into an array of second blocks, wherein araster scan decoding order is defined among the first blocks and thesecond blocks, respectively, the video decoder configured to

determine, depending on a syntax element structure (e.g.,ctb_delay_enabled_flag, min_spatial_segment_delay) of the multi-layervideo data stream, an inter-layer offset for parallel decoding thepictures of the first and second layers by sequentially traversing thefirst and second blocks in a temporally overlapping manner with theinter-layer offset between the traversal of first blocks of pictures ofthe first layer relative to the traversal of spatial second blocks ofpictures of the second layer, measured in units of the first blocks.

A sixty-third aspect concerns the video decoder according to aspect 62,wherein the syntax element structure is a long-term syntax elementstructure, and the video decoder is configured to

perform the determination in advance for a predetermined time period,and

periodically determine, in time intervals smaller than the predeterminedtime period, sizes and locations of the first blocks of the pictures ofthe first layer and the second blocks of the pictures of the secondlayer, and spatial sampling resolutions of the pictures of the firstlayer and the pictures of the second layer, respectively, based onshort-term syntax elements of the of the multi-layer video data stream.

A sixty-fourth aspect concerns the video decoder according to aspect 63,wherein the video decoder supports parallel decoding the multi-layervideo data stream in spatial segments which are sequentially arrangedalong the raster scan decoding order, and into which pictures of thelayers are partitioned, by sequentially traversing the spatial segmentsin a temporally overlapping manner with the inter-layer offset betweenthe traversal of spatial segments of pictures of the first layerrelative to the traversal of spatial segments of pictures of the secondlayer, the video decoder configured to

inspect the long-term syntax element structure of the multi-layer videodata stream so as to

if the long-term syntax element structure (e.g.,ctb_delay_enabled_flag=0, min_spatial_segment_delay≠0) is set to a valueof a first set of possible values, use the value of the long-term syntaxelement structure to determine the inter-layer offset in advance for apredetermined time period measuring the inter-layer offset in units ofspatial segments of the pictures of the first layer and periodicallydetermine, in time intervals smaller than the predetermined time period,sizes and locations of the spatial segments of the pictures of the firstlayer and the spatial segments of the pictures of the second layer, andspatial sampling resolutions of the pictures of the first layer and thepictures of the second layer, respectively, based on short-term syntaxelements of the of the multi-layer video data stream;

If the long-term syntax element structure is set to a value of a secondset (e.g., min_spatial_segment_delay=0) of possible values disjoint tothe first set of possible values, periodically determine, in timeintervals smaller than the predetermined time period, the inter-layeroffset within the predetermined period based on the short-term syntaxelements of the of the multi-layer video data stream; and

If the long-term syntax element is set to a value of a third set (e.g.,ctb_delay_enabled_flag=1, min_spatial_segment_delay≠0) of possiblevalues disjoint to the first and second sets of possible values, performthe determination of the inter-layer offset in units of the first blocksand the periodical determination of the sizes and locations of the firstblocks of the pictures of the first layer and the second blocks of thepictures of the second layer, and spatial sampling resolutions of thepictures of the first layer and the pictures of the second layer,respectively.

A sixty-fifth aspect concerns the video decoder of aspect 63 or 64,wherein the video decoder supports parallel decoding the multi-layervideo data stream in sub-streams out of which pictures of the layers arepartitioned and which are composed of rows of the first and secondblocks, using wavefront parallel processing by sequentially traversingthe substreams in a temporally overlapping manner with an intra-pictureinter-substream delay between the traversal of immediately consecutivesubstreams of the same picture and an inter-layer offset between thetraversal of substreams of pictures of the first layer relative to thetraversal of substreams of pictures of the second layer.

A sixty-sixth aspect concerns the video decoder according to any ofaspects 64 to 65, wherein the long-term syntax element structurecomprises a unit flag (c.p., exemplarily, ctb_delay_enabled_flag) and adelay indicator (c.p., exemplarily, min_spatial_segment_delay), whereinthe video decoder is configured to, in inspecting the long-term syntaxelement structure,

inspect the delay indicator to determine as to whether the same is setto zero or not,

if the delay indicator is set to zero, determining that the value of thelong-term syntax element structure is set to a value of the second set,and

if the delay indicator is set to a non-zero value, use the non-zerovalue to determine the value of the long-term syntax element structureand determine that the value of the long-term syntax element structureis set to a value of the first set if the unit flag is zero, and thatthe value of the long-term syntax element structure is set to a value ofthe third set if the unit flag is one.

A sixty-seventh aspect concerns the video decoder according to any ofaspects 65 to 66, configured to rely on the inter-layer offset incommencing parallel decoding first and second layer.

A sixty-eighth aspect concerns the video decoder according to any ofaspects 62 to 67, configured to check as to whether a numbers of spatialsegments or coding blocks of the first layer have been decodedcompletely with the number s uniquely depending on the syntax elementstructure, and defer commencing decoding the second layer duringdecoding the first layer unless the check reveals that, among spatialsegments or coding blocks of the first layer, at least s have beendecoded completely.

A sixty-ninth aspect concerns the video decoder according to any ofaspects 62 to 68, configured to rely on the inter-layer offset incommencing, and completely processing, parallel decoding first andsecond layer.

A seventieth aspect concerns the video decoder according to any ofaspects 62 to 69, configured to check as to whether a numbers of spatialsegments or coding blocks of the first layer have been decodedcompletely with the numbers uniquely depending on the syntax elementstructure and a number t-1 of already decoded spatial segments or codingblocks of the second layer, and defer commencing decoding ant-th spatialsegment or coding block of the second layer during decoding the firstlayer unless the check reveals that, among spatial segments or codingblocks of the first layer, at least s have been decoded completely.

A seventy-first aspect concerns a method for decoding a multi-layervideo data stream 40 into which a scene is coded in a hierarchy oflayers using inter-layer prediction from a first layer to a secondlayer, the video decoder supporting parallel decoding the multi-layervideo data stream in spatial segments 80 into which pictures 12, 15 ofthe layers are sub-divided, wherein the method comprises

inspecting a long-term syntax element structure (606; e.g.tile_boundaries_aligned_flag) of the multi-layer video data stream so asto

interpreting the long-term syntax element structure assuming a value outof a first possible-values-set (e.g. tile_boundaries_aligned_flag=1), asa guarantee that, during a predetermined time period 608, the pictures15 of the second layer are subdivided so that borders between thespatial segments of the pictures of the second layer overlay everyborder of the spatial segments of the pictures 12 of the first layer andperiodically determining, in time intervals 604 smaller than thepredetermined time period, the subdivision of the pictures of the firstlayer and the second layer into the spatial segments based on short-termsyntax elements (602; e.g. column_width_minus1[i] andcolumn_width_minus1[i]) of the multi-layer video data stream, and

if the long-term syntax element structure has assumed a value out of asecond possible-values-set (e.g. tile_boundaries_aligned_flag=0),periodically determining, in the time intervals smaller than thepredetermined time period, the subdivision of the pictures of the layersinto the spatial segments from the short-term syntax elements of themulti-layer video data stream such that, at least for a first possiblevalue of the short-term syntax elements, there exists a border betweenthe spatial segments of the pictures of the second layer not overlayingany of the borders of the spatial segments of the first layer, and, atleast for a second possible value of the short-term syntax elements,borders between the spatial segments of the pictures of the second layeroverlay every border of the spatial segments of the first layer.

A seventy-second aspect concerns a method for encoding a scene into amulti-layer video data stream in a hierarchy of layers using inter-layerprediction from a first layer to a second layer so that the multi-layervideo data stream is decodable in parallel in spatial segments intowhich pictures of the layers are sub-divided, wherein the methodcomprises

inserting a long-term syntax element structure 606 and short-term syntaxelements 602 into the multi-layer video data stream, the short-termsyntax elements defining, in time intervals, the subdivision of thepictures of the first layer and the second layer into the spatialsegments, and

switching between setting the long-term syntax element structure to

-   -   a value out of a first possible-values-set with, during a        predetermined time period 608 greater than the time intervals,        setting the short-term syntax elements to a proper subset out of        a set of possible settings, the proper subset being chosen so        that, during the predetermined time period, the pictures of the        second layer are subdivided so that borders between the spatial        segments of the pictures of the second layer overlay every        border of the spatial segments of the first layer, or    -   a value out of a second possible-values-set with, during the        predetermined time period, setting the short-term syntax        elements to any of the set of possible settings, the set of        possible settings encompassing at least one setting according to        which a border exists between the spatial segments of the        pictures of the second layer not overlaying any of the borders        of the spatial segments of the first layer, and at least another        setting according to which borders between the spatial segments        of the pictures of the second layer overlay every border of the        spatial segments of the first layer.

A seventy-third aspect concerns a method for decoding a spatiallyscalable bitstream 40 into which a picture is coded in different spatiallayers and, for at least one of the spatial layers, in first spatialsegments, wherein the method comprises

upsampling a picture 12 of a first spatial layer so as to obtain anupsampled reference picture and predict a picture 15 of a second spatiallayer using the upsampled reference picture, wherein the method fordecoding is responsive to a syntax element (616; e.g.,independent_tile_upsampling_idc) in the spatially scalable bitstream soas to, depending on the syntax element, interpolate 620 the picture ofthe first spatial layer such that any partition of a partitioning 622 ofthe upsampled reference picture, which depends on the first spatialsegments, is independent from portions of the picture of the firstspatial layer covered by any of the other partitions of thepartitioning, or such that any partition of the partitioning 622 of theupsampled reference picture is dependent on portions of the picture ofthe first spatial layer covered by another partition of the partitioningspatially neighbouring the respective partition.

A seventy-fourth aspect concerns a method for encoding a picture into aspatially scalable bitstream in different spatial layers and, for atleast one of the spatial layers, in first spatial segments, wherein themethod comprises

upsampling a picture of a first spatial layer so as to obtain anupsampled reference picture and predict a picture of a second spatiallayer using the upsampled reference picture, wherein the methodcomprises setting and inserting a syntax element 606 into the spatiallyscalable bitstream and, depending on the syntax element, interpolatingthe picture of the first spatial layer such that any partition of apartitioning of the upsampled reference picture, which depends on thefirst spatial segments, is independent from portions of the picture ofthe first spatial layer covered by any of the other partitions of thepartitioning, or such that any partition of the partitioning of theupsampled reference picture is dependent on portions of the picture ofthe first spatial layer covered by another partition of the partitioningspatially neighbouring the respective partition.

A seventy-fifth aspect concerns a method for decoding a multi-layervideo data stream into which a scene is coded in a hierarchy of layersusing inter-layer prediction from a first layer to a second layer, thevideo decoder supporting parallel decoding the multi-layer video datastream in spatial segments into which pictures of the layers arepartitioned by sequentially traversing the spatial segments in atemporally overlapping manner with an inter-layer delay between thetraversal of spatial segments of pictures of the first layer relative tothe traversal of spatial segments of pictures of the second layer, themethod comprises

inspecting a long-term syntax element structure (642; e.g.min_spatial_segment_delay) of the multi-layer video data stream so as to

if the long-term syntax element structure (e.g.min_spatial_segment_delay≠0) is set to a value of a first set ofpossible values, using the value of the long-term syntax element todetermine the inter-layer offset in advance for a predetermined timeperiod and periodically determining, in time intervals smaller than thepredetermined time period, sizes and locations of the spatial segmentsof the pictures of the first layer and the spatial segments of thepictures of the second layer, and spatial sampling resolutions of thepictures of the first layer and the pictures of the second layer,respectively, based on short-term syntax elements 602 of the of themulti-layer video data stream;

if the long-term syntax element is set to a value of a second set (e.g.min_spatial_segment_delay=0) of possible values disjoint to the firstset of possible values, periodically determining, in time intervalssmaller than the predetermined time period, the inter-layer offsetwithin the predetermined period based on the short-term syntax elementsof the of the multi-layer video data stream.

A seventy-sixth aspect concerns a method for encoding a scene into amulti-layer video data stream in a hierarchy of layers using inter-layerprediction from a first layer to a second layer such that themulti-layer video data stream is decodable in spatial segments intowhich pictures of the layers are partitioned by sequentially traversingthe spatial segments in a temporally overlapping manner with aninter-layer offset between the traversal of spatial segments of picturesof the first layer relative to the traversal of spatial segments ofpictures of the second layer, the method comprising

Inserting and setting a long-term syntax element structure(min_spatial_segment_delay) and short-term syntax elements into themulti-layer video data stream, the short-term syntax elements of the ofthe multi-layer video data stream defining, in a periodical basis, intime intervals, sizes and locations of the spatial segments of thepictures of the first layer and the spatial segments of the pictures ofthe second layer, and spatial sampling resolutions of the pictures ofthe first layer and the pictures of the second layer, respectively,

wherein the method comprises switching between setting

the long-term syntax element structure (min_spatial_segment_delay≠0) toa value of a first set of possible values, with the value signaling theinter-layer offset for a predetermined time period greater than the timeintervals, with, for the predetermined time period, setting theshort-term syntax elements to a proper subset out of a set of possiblesettings, the proper subset being chosen so that, during thepredetermined time period, the sizes and the locations of the spatialsegments of the pictures of the first layer and the spatial segments ofthe pictures of the second layer, and the spatial sampling resolutionsof the pictures of the first layer and the pictures of the second layer,respectively, enable decoding the multi-layer video data stream bysequentially traversing the spatial segments in the temporallyoverlapping manner with an actual inter-layer offset which is smallerthan or equal to the inter-layer offset signaled by the long-term syntaxelement, between the traversal of spatial segments of pictures of thefirst layer relative to the traversal of spatial segments of pictures ofthe second layer;

the long-term syntax element to a value of a second set(min_spatial_segment_delay=0) of possible values disjoint to the firstset of possible values, with, during the predetermined time period,setting the short-term syntax elements to any of the set of possiblesettings, the set of possible settings encompassing at least one settingaccording to which the sizes and the locations of the spatial segmentsof the pictures of the first layer and the spatial segments of thepictures of the second layer, and the spatial sampling resolutions ofthe pictures of the first layer and the pictures of the second layer,respectively, disable decoding the multi-layer video data stream bysequentially traversing the spatial segments in the temporallyoverlapping manner with an actual inter-layer offset which is smallerthan or equal to the inter-layer offset signaled by the long-term syntaxelement, between the traversal of spatial segments of pictures of thefirst layer relative to the traversal of spatial segments of pictures ofthe second layer, and at least another setting according to which thesizes and the locations of the spatial segments of the pictures of thefirst layer and the spatial segments of the pictures of the secondlayer, and the spatial sampling resolutions of the pictures of the firstlayer and the pictures of the second layer, respectively, enabledecoding the multi-layer video data stream by sequentially traversingthe spatial segments in the temporally overlapping manner with an actualinter-layer offset which is smaller than or equal to the inter-layeroffset signaled by the long-term syntax element, between the traversalof spatial segments of pictures of the first layer relative to thetraversal of spatial segments of pictures of the second layer.

A seventy-seventh aspect concerns a method for processing a multi-layervideo data stream into which a scene is coded in layers so that, in eachlayer, the scene is coded in a different operational point of ascalability space spanned by scalability dimensions, wherein themulti-layer video data stream is composed of first NAL units each ofwhich as associated with one of the layers, and second NAL unitsinterspersed within the first NAL units and presenting generalinformation on the multi-layer video data stream, the method comprising

inspecting a type indicator field (696, e.g.dedicated_scalability_ids_flag) in the second NAL units;

if the type indicator field has a first state (e.g.dedicated_scalability_ids_flag=0), reading mapping information (e.g.layer_id_in_nuh[i], dimension_id[i][j]) mapping possible values of alayer indicator field (e.g. layer_id) in the first NAL unit headers tothe operational points from the second NAL units and associate the firstNAL units with the operational points in the first NAL units via thelayer indicator field and the mapping information;

if the type indicator field has a second state(dedicated_scalability_ids_flag=1), associating the first NAL units withthe operational points by splitting the layer indicator field in thefirst NAL units into more than one portion and locating the operationalpoint of the first NAL units by using the values of the portions ascoordinates of a vector within the scalability space.

A seventy-eighth aspect concerns a method for encoding a scene into amulti-layer video data stream in layers so that, in each layer, thescene is coded in a different operational point of a scalability spacespanned by scalability dimensions, wherein the multi-layer video datastream is composed of first NAL units each of which as associated withone of the layers, and second NAL units interspersed within the firstNAL units and presenting general information on the multi-layer videodata stream, the method comprising

inserting a type indicator field into the second NAL units and switchingbetween setting

the type indicator field such that same has a first state, withinserting mapping information mapping possible values of a layerindicator field in the first NAL unit headers to the operational pointsinto the second NAL units and setting the layer indicator filed in thefirst NAL units such that the first NAL units' operational points areassociated with the respective layer indicator field via the mappinginformation;

the type indicator field such that same has a second state(dedicated_scalability_ids_flag=1), with setting the layer indicatorfield in the first NAL units by splitting the layer indicator field inthe first NAL units into more than one portion and setting the more thanportion so that the values of the portions correspond to coordinates ofa vector within the scalability space, pointing to the operational pointassociated with the respective first NAL unit.

A seventy-ninth aspect concerns a method for subjecting a multi-layervideo data stream into which a scene is coded in layers, to decoding bya multi-standard multi-layer decoder, wherein the multi-layer video datastream is composed of NAL units each of which is associated with one ofthe layers, wherein the layers are associated with different codecs sothat, for each layer, the NAL units associated with the respective layerare coded using the codec with which the respective layer is associated,the method comprises

identifying, for each NAL unit, as to which codec same is associatedwith, and

handing over the NAL units of the multi-layer video data stream to themulti-standard multi-layer decoder which uses inter-layer predictionbetween layers associated with different codecs to decode themulti-layer video data stream.

An eightieth aspect concerns a method for decoding a multi-layer videodata stream into which a scene is coded in a hierarchy of layers usinginter-layer prediction from portions of a first layer to co-locatedportions of a second layer, wherein pictures of the first layer aresub-divided into an array of first blocks and pictures of the secondlayer are sub-divided into an array of second blocks, wherein a rasterscan decoding order is defined among the first blocks and the secondblocks, respectively, the method comprises

determining, depending on a syntax element structure (e.g.,ctb_delay_enabled_flag, min_spatial_segment_delay) of the multi-layervideo data stream, an inter-layer offset for parallel decoding thepictures of the first and second layers by sequentially traversing thefirst and second blocks in a temporally overlapping manner with theinter-layer offset between the traversal of first blocks of pictures ofthe first layer relative to the traversal of spatial second blocks ofpictures of the second layer, measured in units of the first blocks.

An eighty-first aspect concerns a computer program having a program codefor performing, when running on a computer, a method according to any ofaspects 71 to 80.

While this invention has been described in terms of several advantageousembodiments, there are alterations, permutations, and equivalents whichfall within the scope of this invention. It should also be noted thatthere are many alternative ways of implementing the methods andcompositions of the present invention. It is therefore intended that thefollowing appended claims be interpreted as including all suchalterations, permutations, and equivalents as fall within the truespirit and scope of the present invention.

1. A video decoder for decoding a multi-layer video data stream intowhich a video is coded in layers, wherein the video decoder isconfigured for decoding the multi-layer video data stream in accordancewith spatial segments derived by sub-dividing pictures of the video, thevideo decoder comprising: an entropy decoder configured for, withrespect to a spatial segment of a picture related to a first layer or asecond layer of the layers, entropy decoding data from the multi-layervideo data stream to obtain residual data and a coding parameter; apredictor configured for generating a prediction signal associated withthe spatial segment based at least on the coding parameter; and acombiner configured for combining the prediction signal and the residualdata to reconstruct a portion of the picture, wherein the video decoderis configured for determining subdivision of pictures of the first andsecond layers by decoding a long-term syntax element and a short-termsyntax element from the multi-layer video data stream, wherein thelong-term syntax element having a first value indicates that a pictureof the second layer is subdivided in a manner that at least one of thespatial segments of the picture of the second layer is spatially alignedwith a corresponding one of the spatial segments of a picture of thefirst layer, and determining the subdivision of the pictures of thefirst layer and the second layer into their respective spatial segmentsbased on the short-term syntax element.
 2. The video decoder accordingto claim 1, wherein the video decoder is configured for decoding thepictures of the first or second layer using intra-picture spatialprediction with disrupting the intra-picture spatial prediction for eachspatial segment at a boundary of the respective spatial segment, ordecoding the pictures of the layers using intra-picture spatialprediction by decoding the spatial segments of a picture of the firstlayer in parallel with supporting the intra-picture spatial predictioncrossing boundaries of the spatial segments of the picture of the firstlayer and obeying a decoding delay between the decoding of the spatialsegments of the picture of the first layer, and by decoding the spatialsegments of a picture of the second layer in parallel with supportingthe intra-picture spatial prediction crossing boundaries of the spatialsegments of the picture of the second layer and obeying a decoding delaybetween the decoding of the spatial segments of the picture of thesecond layer.
 3. The video decoder according to claim 1, wherein thespatial segments into which the pictures are sub-divided include tiles.4. The video decoder according to claim 3, wherein the video decoder isconfigured for decoding the pictures of the layers using intra-picturespatial prediction with disrupting the intra-picture spatial predictionfor each tile at a boundary of the respective tile.
 5. The video decoderaccording to claim 1, wherein each spatial segment of the picture of thefirst layer is made up of exactly n spatial segments of the picture ofsecond layer with n depending on the value of the long-term syntaxelement, and if the long-term syntax element has a second value, thevideo decoder is configured for determining, in time intervals smallerthan a predetermined time period, an inter-layer offset within thepredetermined time period based on the short-term syntax element.
 6. Thevideo decoder according to claim 1, wherein the video decoder isconfigured to start decoding the second layer of the multi-layer videodata stream depending on the value of the long-term syntax element. 7.The video decoder according to claim 1, wherein the video decoder is ahybrid video decoder.
 8. The video decoder according to claim 1, whereinthe long-term syntax element having the first value indicates that,during a predetermined time period, the picture of the second layer issubdivided such that borders between the spatial segments of the pictureof the second layer overlay every border of the spatial segments of thepicture of the first layer.
 9. A video encoder for encoding a video intoa multi-layer video data stream in layers so that the multi-layer videodata stream is decodable in accordance with spatial segments derived bysub-dividing pictures of the video, the video encoder comprising: apredictor configured for generating, with respect to a spatial segmentof a picture related to a first layer or a second layer of the layers, aprediction signal based on a coding parameter; a residual determinerconfigured for determining residual data associated with the spatialsegment of the picture; and an entropy encoder configured for entropyencoding the coding parameter and the residual data into the multi-layervideo data stream, wherein the video encoder is configured for signalingsubdivision of pictures of the first and second layers by inserting along-term syntax element and a short-term syntax element into themulti-layer video data stream, the long-term syntax element having afirst value indicates that a picture of the second layer is subdividedin a manner that at least one of the spatial segments of the picture ofthe second layer is spatially aligned with a corresponding one of thespatial segments of the picture of the first layer, and the short-termsyntax element defining the subdivision of the pictures of the firstlayer and the second layer into their respective spatial segments. 10.The video encoder according to claim 9, wherein the video encoder isconfigured for encoding the pictures of the first or second layer usingintra-picture spatial prediction with disrupting the intra-picturespatial prediction for each spatial segment at a boundary of therespective spatial segment, or encoding the pictures of the layers usingintra-picture spatial prediction and entropy encoding with adaptingentropy context probabilities, by supporting the intra-picture spatialprediction crossing boundaries of the spatial segments of a picture ofthe first layer and initializing the entropy context probabilities forthe entropy encoding of the subsets of the spatial segments of thepicture of the first layer individually or under adopting the entropycontext probabilities of a, in accordance with an order among thesubsets, previous subset of the spatial segments of the picture of thefirst layer at an intermediately adapted state, and by supporting theintra-picture spatial prediction crossing boundaries of the spatialsegments of a picture of the second layer and initializing the entropycontext probabilities for the entropy encoding of the subsets of thespatial segments of the picture of the second layer individually orunder adopting the entropy context probabilities of a, in accordancewith an order among the subsets, previous subset of the spatial segmentsof the picture of the second layer at an intermediately adapted state.11. The video encoder according to claim 9, wherein the spatial segmentsinclude tiles.
 12. The video encoder according to claim 11, wherein thevideo encoder is configured for encoding the pictures of the layersusing intra-picture spatial prediction with disrupting the intra-picturespatial prediction for each tile at a boundary of the respective tile.13. The video encoder according to claim 9, wherein when the long-termsyntax element is set to the first value, each spatial segment of thepicture of the first layer is made up of exactly n spatial segments ofthe picture of second layer with n depending on the value of thelong-term syntax element.
 14. The video encoder according to claim 9,wherein the long-term syntax element having the first value indicatesthat, during a predetermined time period, the picture of the secondlayer is subdivided such that borders between the spatial segments ofthe picture of the second layer overlay every border of the spatialsegments of the picture of the first layer.
 15. A non-transitorycomputer-readable medium for storing video data, comprising: a datastream stored in the non-transitory computer-readable medium andcomprising a multi-layer video data stream into which a video is codedin layers, the decoding of the multi-layer video data stream beingperformed in accordance with spatial segments derived by sub-dividingpictures of the video by executing operations using a processor, theoperations including: entropy decoding data from the multi-layer videodata stream to obtain, with respect to a spatial segment of a picture ofa first layer or a second layer of the layers, residual data and acoding parameter; generating a prediction signal associated with thespatial segment based at least on the coding parameter; combining theprediction signal and the residual data to reconstruct a portion of thepicture; and determining subdivision of pictures of the first and secondlayers by decoding a long-term syntax element and a short-term syntaxelement from the multi-layer video data stream, wherein the long-termsyntax element having a first value indicates that a picture of thesecond layer is subdivided in a manner that at least one of the spatialsegments of the picture of the second layer is spatially aligned with acorresponding one of the spatial segments of a picture of the firstlayer, and determining the subdivision of the pictures of the firstlayer and the second layer into their respective spatial segments basedon the short-term syntax element.
 16. The non-transitorycomputer-readable medium according to claim 15, the operations furtherincluding: decoding the pictures of the first or second layer usingintra-picture spatial prediction with disrupting the intra-picturespatial prediction for each spatial segment at a boundary of therespective spatial segment, or decoding the pictures of the layers usingintra-picture spatial prediction by decoding the spatial segments of apicture of the first layer in parallel with supporting the intra-picturespatial prediction crossing boundaries of the spatial segments of thepicture of the first layer and obeying a decoding delay between thedecoding of the spatial segments of the picture of the first layer, andby decoding the spatial segments of a picture of the second layer inparallel with supporting the intra-picture spatial prediction crossingboundaries of the spatial segments of the picture of the second layerand obeying a decoding delay between the decoding of the spatialsegments of the picture of the second layer.
 17. The non-transitorycomputer-readable medium according to claim 15, wherein the spatialsegments into which the pictures are sub-divided include tiles.
 18. Thenon-transitory computer-readable medium according to claim 17, theoperations further including decoding the pictures of the layers usingintra-picture spatial prediction with disrupting the intra-picturespatial prediction for each tile at a boundary of the respective tile.19. The non-transitory computer-readable medium according to claim 15,wherein each spatial segment of the picture of the first layer is madeup of exactly n spatial segments of the picture of second layer with ndepending on the value of the long-term syntax element.
 20. Thenon-transitory computer-readable medium according to claim 15, whereinthe long-term syntax element having the first value indicates that,during a predetermined time period, the picture of the second layer issubdivided such that borders between the spatial segments of the pictureof the second layer overlay every border of the spatial segments of thepicture of the first layer.